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We study a degenerate two-species gas of bosonic atoms interacting tiirougii a p-wave Feslibacii 
resonance as for example realized in a *^Rb-*^Rb mixture. We show that in addition to a conventional 
atomic and a p-wave molecular spinor-1 superfluidity at large positive and negative detunings, 
respectively, the system generically exhibits a finite momentum atomic-molecular superfluidity at 
intermediate detuning around the unitary point. We analyze the detailed nature of the corresponding 
phases and the associated quantum and thermal phase transitions. 



I. INTRODUCTION 
A. Background and Motivation 

Since the experimental realization of Bose-Einstein 
condensation (BEC) in trapped alkali-metal-atom 
gasesii^, the resulting burgeoning field of degenerate 
atomic gases has seen an ever-expanding research activ- 
ity. It has been fueled by the steady advances in new 
experimental techniques to control and interrogate the 
continually growing class of degenerate atomic systems. 
A Feshbach resonance (FR) has been one of these excep- 
tionally fruitful experimental "knobs" that lends exquisite 
tunability (via magnetic field) of interactions in the ul- 
tracold atomic gases. For fermionic trapped gases, it en- 
abled a realization of a fermionic atom-paired s-wave su- 
perfluidity and exploration of its BEC-BCS crossover and 
resonant universality^^—. 

Motivated by the demonstration of p-wave FR in ""^K 
and ^Li, p-wave paired fermionic superfluidity has also 
been extensively explored theoretically^i"— , predicting 
to exhibit an even richer phenomenology. A recent lab- 
oratory production of p-wave Feshbach molecule a ' ""'"- 
shows considerable promise toward a realization of p- 
wave fermion-paired superfluidity and the associated rich 
phenomenology^!, though substantial challenges of sta- 
bility remain2^i21. 

The bosonic counterparts have also been extensively 
explored and in fact in the s-wave FR case of *^Rb~ 
predate recent fermionic developments. As was recently 
emphasized^^Ti^l, in contrast to their fermionic analogs, 
which undergo a smooth BEC-BCS crossover, resonant 
bosonic gases are predicted to exhibit magnetic-field- 
and/or temperature-driven sharp phase transitions be- 
tween distinct molecular and atomic superfluid phases. 
One serious impediment to a laboratory realization of 
this rich physics is the predicted^i^ and observed'^'* in- 
stabilities of a resonantly attractive Bose gas sufficiently 
close to a FR. Nevertheless, a number of features of 
the phase diagram are expected to be exhibited away 
from the resonance and/or reflected in the nonequilib- 
rium phenomenology (before the onset of the instability) 
of a resonant Bose gas. Furthermore, recent extension 
to an s-wave resonant Bose gas in an optical lattice^i^ 
demonstrated the stabilization through a quantum Zeno 



mechanism proposed by Rempe^, which dates back to 
Bethe's^ analysis of the triplet linewidth in hydrogen. 
The predictions22i"— have been supported by recent 
density matrix renormalization group^^, exact diagonal- 
ization^^, and quantum Monte Carle studies, as well 
extensions to two species^. 



Along with the ubiquitous s-wave resonances, recent 
experiments on a ^^Rb-*^Rb mixture have demonstrated 
an interspecies p-wave FR at B = 257.8 G^i^. Although 
the consequences of this two-body p-wave resonance on 
the degenerate many-body state of such a gas mixture 
has not been explored experimentally, it provided the 
main motivation for our recent^ and present studies. We 
note that closely related studies of BEC in p (and higher) 
bands in optical lattices have been carried out in Refs.H. 

m. 



The rest of the paper is organized as follows. We con- 
clude the Introduction with a summary of our main re- 
sults and their experimental implications. In Sec. HIl we 
introduce a microscopic two-channel p-wave FR model 
for a description of a two-component Bose g for 
example realized by a *^Rb-*^Rb mixture. Having re- 
lated the parameters of the model to two-body scatter- 
ing experiments on a dilute gas, in Sec. IIIII we present a 
general symmetry-based discussion of phases and associ- 
ated phase transitions expected in such an atomic gas at 
finite density. In Sec. IIVI by minimizing the correspond- 
ing imaginary-time coherent state action, we map out a 
generic mean-field phase diagram for this system. In Sec. 
IVl we supplement this Landau analysis with a derivation 
of the corresponding Goldstone-mode Lagrangians and 
extract from them the low-energy elementary excitations 
and dispersions characteristic of each phase. The true 
(beyond-mean-field) nature of the quantum and thermal 
phase transitions is discussed in Sec. I VII In Sec. I VIII we 
study the topological defects, vortices and domain walls, 
in each of the phases. We make a more direct contact 
with cold-atom experiments in Sec. IVIIII bv using a lo- 
cal density approximation (LDA) to include the effects of 
the trapping potential. We close with a brief summary 
in Sec. El 
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B. Summary of results 

Before turning to the analysis of the system, we present 
the main predictions of our work, a small subset of 
which was previously reported in a Letter—. Our key 
results are summarized by a FR temperature-detuning 
phase diagram, illustrated in Fig. [T] and by the prop- 
erties of the corresponding phases and transitions. We 




FIG. 1: Schematic temperature-detuning phase diagram for 
a balanced two-species p-wave resonant Bose gas. As il- 
lustrated, it exhibits atomic (ASF), molecular (MSF), and 
atomic-molecular (AMSF) superfluid phases. The AMSF 
state is characterized by a p-wave, molecular, and finite- 
momentum Q (see Fig. [2]) atomic superfluidity. 

find that in addition to the normal (i.e., non-superfluid) 
high-temperature phase, the p-wave Feshbach-resonant 
two-component balanced Bose gas (e.g., equal mixture of 
^^Rb and ^''Rb atoms) generically exhibits three classes of 
superfluid phases: atomic (ASF), molecular (MSF), and 
atomic-molecular (AMSF) condensates, where atoms, p- 
wave molecules, and both are Bose-Einstein-condensed, 
respectively. Our most interesting finding is that the 
AMSF phase, sandwiched between (large positive detun- 
ing) ASF and (large negative detuning) MSF phases, is 
necessarily a finite- momentum Q spinor superfluid, akin 
to (but distinct from) a supersolid^^^^^^i. It is character- 
ized by a momentum hQ, with its magnitude 

Q = arriy/n^ ^ Jp£nm < ^/%/^, (1.1) 

tunable with a magnetic held [via FR detuning, v that 
primarily enters through the molecular condensate den- 
sity nm(i')], with a, m, and 7p, respectively, the FR 
coupling, atomic mass, average atom spacing, and a di- 
mensionless measure of FR width^^. 

As illustrated in the phase diagram (Fig. [T]), the ASF 
appears at a large positive detuning (weak FR attraction) 
and low temperature, where one of the three combina- 
tions (ASFi, ASF2, ASF12) of the s^Rb and ^'^Rb atoms 
is Bose-Einstein-condensed into a conventional, uniform 
superfluid and the p-wave *^Rb-^^Rb molecules are en- 
ergetically costly and therefore appear only as gapped 
excitations. 




FIG. 2: (Color online) Schematic momentum Q(i^) character- 
istic of the AMSF (polar) state, ranging between zero and the 
p-wave FR width-dependent value. 



In the complementary regime of a large negative detun- 
ing, the attraction between two flavors of atoms is suffi- 
ciently strong so as to bind them into tight p-wave hetero- 
molecules (e.g., ^^Rb-®''Rb molecule), which at low tem- 
perature condense into a p-wave superfluid, with atoms 
in the species-balanced case existing only as gapped ex- 
citations. In this tight-binding molecular regime the gas 
reduces to a well-explored system of a spinor- 1 conden- 
sate^"—, with the spinor corresponding to the relative 
orbital angular momentum £ = 1 of the two constituent 
atoms of the p-wave molecule. Thus, for negative detun- 
ing we predict the existence of £z =0 "polar" (MSFp) 
and £2 = ±1 "ferromagnetic" (MSFfm) molecular p-wave 
superfluid phases, with their relative stability determined 
by the ratio 09/02 of molecular spin-0 (oq) to molecular 
spin-2 (02) scattering lengths. We flnd that this ratio and 
therefore the first-order MSFp-MSFfm transition are, in 
turn, controlled with the p-wave FR detuning v, or equiv- 
alently, the atomic p-wave scattering volume v ~ 1/:^, 
tunable with a magnetic field. 

We emphasize that (in contrast to the s-wave case22r^) 
because a p-wave resonance does not couple a uniform 
atomic condensate to the molecular one, a p-wave molec- 
ular condensate is not automatically induced inside the 
ASF state. 

The most distinctive signatures of these superfluids 
should be directly detectable via time-of-flight shadow 



mm 

FIG. 3: (Color online) A cartoon of a p-wave molecule decay- 
ing into two oppositely moving species of atoms, illustrating 
a resonant mechanism for a finite-momentum Q atomic su- 
perfluidity (indicated by wavy lines) in the AMSF phase. 
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images, with the ASF exhibiting an atomic condensate 
peak and the MSF displaying a p-wave molecular one. 
At higher densities in a trap, the bulk phase diagram as 
a function of the chemical potential (see Fig. and [^T|) 
translates into shell structure of distinct phases, which 
we estimated within the LDA^^i^. 

In addition to these fairly conventional uniform atomic 
and molecular BECs, for intermediate detuning around 
a unitary point we predict the existence of AMSFp and 
AMSFfni phases, characterized by a finite momentum Q 
atomic condensate^i^, that is a superposition of the two 
atomic species. Such a generically supersolid stated"— 
is always accompanied by a p-wave molecular conden- 
sate, concomitantly induced through the p-wave FR in- 
teraction. In addition to exhibiting an ofi-diagonal long- 
range order (ODLRO) of an ordinary superfluid the two 
AMSFp states (distinguished by the polar versus ferro- 
magnetic nature of their p-wave molecular condensates) 
spontaneously partially break orientational and transla- 
tional symmetries, akin to polar and smectic liquid crys- 
tals'^- and the putative Fulde-Ferrell-Larkin-Ovchinnikov 
states of imbalanced paired fermionsSJS. 

As iUustrated in Fig. gl in the AMSFp state, Q aligns 
along the quantization axis along which the molecular 
condensate has a zero projection of its internal i — 1 
angular momentum. For the case of the AMSFfm state, 
Q lies in the otherwise isotropic plane, transverse to the 
p-wave molecular condensate axis, as illustrated in Fig.[S] 

In the narrow FR approximation we find that the 
AMSFpjm states are collinear, characterized by a sin- 
gle Q of a Fulde-Ferrell-like fornt^^, as opposed to a + Q 
and — Q Larkin-Ovchinnikov-like^^ or other more com- 
plicated crystalline forms, found in imbalanced paired 
fermionic systems^^"— . However, because the detailed 
spatial structure of the AMSFfm (but not the AMSFp 
state) sensitively depends on the interactions (since it 
spontaneously breaks symmetry transverse to the = 1 
axis), we do not exclude a more general lattice structure 
in a more generic beyond-mean-field model, which is best 
analyzed numerically. 

The phase boundaries between this rich variety of 
phases can be calculated for a narrow FR and in a dilute 
Bose-gas limit, but are notoriously difficult to estimate 
in a strongly interacting system, where they can only be 
qualitatively estimated within a mean-field analysis. In 
the former case the zero-temperature phase boundaries 
are given by critical detunings: 



,MSF„ -AMSFp 



,AMSF„-ASF 



= - (.91 + .92 - 2ga, 



ft2 



)n,,„, (1.2a) 



{2X - gam + -;rr^)na, for 52 < 0, 



(1.2b) 



with similar expressions for transitions out of the ferro- 
magnetic phases, which can be found in Eq. (|4.35|) . Here 
(jj's and Ai's are molecular and atomic two-body interac- 
tion pseudopotentials, respectively related to the back- 



ground molecular (cq^ and a^^) and atomic scattering 
lengths {all, ^22^ ^^'^ '^i2)- 

As any neutral superfluid, ASF, MSF, and AMSF are 
each characterized by Bogoliubov modes, illustrated in 
Figs. [131 EH [13 and [TH with long-wavelength acoustic 
"sound" dispersions. 



rhk, 



(1.3) 



where (with a =ASFi,2a2, MSFp,fm, AMSFp^^) are 
the associated sound speeds with standard Bogoliubov 
form Co- ~ ^ g„n„ jlm. In each of these SF states one 
Bogoliubov mode (and only one in the ASF^ states) cor- 
responds to the overall condensate phase fluctuations. 
In addition, the MSFp exhibits two degenerate "trans- 
verse" Bogoliubov orientational acoustic modes. The 
MSFfm is also additionally characterized by one "ferro- 
magnetic" spin- wave mode, ~ fc^ g^j^^j gapped 
mode, consistent with the characteristics of a conven- 
tional spinor-1 condensate^^iS^. 

Because MSFp f^'s are paired MSFs, they also exhibit 
gapped single atomlike quasiparticles (akin to Bogoli- 
ubov excitations in a fermionic paired BCS state) that 
do not carry a definite atom number. These single- 
particle excitations are "squeezed" by the presence of 
the molecular condensate, offering a mechanism to re- 
alize atomic squeezed statesSi, which can be measured 
by interference experiments, similar to those reported in 
Ref. The low-energy nature of these single-atom ex- 
citations is guaranteed by the vanishing of the gap at 

MSFp.t,„-AMSFp,f,„ 



the MSF-AMSF transition at 
<sf(^c) = 0. 



with 



p.fm 



We also note that inside the MSFp^m, for v > v, 
-(ffp,fm + C'p,finTOa^/fi^)nm, where Cpjm = 2,1 for polar 
and ferromagnetic phases, respectively, the minimum of 
the single-atom excitations (that for < i^* is at fc = 0) 
shifts to a finite momentum, k k, Q. This is a precur- 
sor of the atomic gap-closing MSF-AMSF transition at 
j^MSF-AMSF^ where atoms also Bose condense at finite 
momentum Q. 

We predict that in addition to the conventional Bogoli- 
ubov superfluid mode associated with the phase common 
to the atomic and molecular condensates, the AMSF also 
exhibits a Goldstone mode corresponding to the fluctu- 
ation of a relative phase between the two atomic con- 
densate components. Furthermore, a spatially periodic, 
collinear AMSF state, characterized by at least ± Q mo- 
menta (but not just single Q) further exhibits the con- 
densate phonon mode u corresponding to the difference 
between phases of the ±Q condensate components, akin 
to the Larkin-Ovchinnikov state^i^i^. 

For the single Q AMSF states, we predict the smec- 
ticlike "phonon" spectra in the polar and ferromagnetic 
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cases, 



a;AMSFp(fc) - ^{Bkl + Kki)/x- 



(1.4a) 



(1.4b) 

as well as the conventional Bogoliubov modes associ- 
ated with superfluid order, and an orientational mode 
'^fm' associated with orientational symmetry breaking in 
AMSFfm 



X+m 



(fc)=1 



jp [Bki + fc2(ii:^fc2 + Kykl) 



where B 



K = 



K = ^ 2Ky 

■lib ■ y 

and xl^ = 5(A- A12). 



^ J 



(1.5a) 



(1.5b) 



4m 



Having summarized the results of our study, we next 
turn to the definition of the two-component p-wave reso- 
nant Bose-gas model, followed by its detailed analysis. 



II. MODEL 

We study a gas mixture of two distinguishable bosonic 
atoms (e.g., ^^Rb, ^''Rb)^'^, created by field operators 

■^^.(r) = (^tpl{r) , ^pl{r)^ and interacting through a p-wave 

FR associated with a tunable "closed"-channel bound 
state. The corresponding p-wave {£ = 1) closed-channel 
hetero- molecule (e.g., ^'^Rb-^^Rb) is created by a Carte- 
sian vector field operator 0^(r) = (0j,, 0^^, 0j), related 

to — {<j>l. ± i(/)p/\/2, — <j>l operators, which create 
closed-channel molecules in the £z — ±1,0 eigenstates, re- 
spectively. This system is governed by a grand-canonical 
Hamiltonian density (with h ~ I throughout). 



n = 



(7=1,2 



Iplialpa + ■ Co ■ 4) + % 



bg 



(2.1) 



where single-particle atomic and molecular Hamiltonians 
are given by 



2 m 



Mo 



1 

4m 



with the effective molecular chemical potential. 

Aim = Ml + - 



(2.2a) 
(2.2b) 

(2.3) 



adjustable by a magnetic-field-dependent detuning i/, 
the latter being the rest energy of the closed-channel 



molecule relative to a pair of open-channel atoms. For 
simplicity we have taken atomic masses to be identical 
(a good approximation for the ^^Rb-®''Rb mixture that 
we have in mind) and will focus on the balanced case 
of Ml = M2 = M: with fi fixing the total number of *^Rb 
and ®^Rb atoms, whether in the (open-channel) atomic or 
(closed-channel) molecular form. The FR interaction en- 
codes a coherent interconversion between a pair of open- 
channel atoms 1,2 (in a singlet combination of 1, 2 labels, 
as required by bosonic statistics) and a closed-channel p- 
wave molecule, with amplitude 

The FR coupling a and detuning v are fixed experi- 
mentally through measurements of the low-energy two- 
atom p-wave scattering amplitude^^i^, 



fp{k) 



hkok"^ — ik^ ' 



(2.4) 



where v is the scattering volume (tunable via magnetic 
field dependent detuning i/) and feg (negative for the FR 
case) is the characteristic wave vectori^iiSl, 



ko 



6n 

mcP- 
12tt 



(I + C2), 



(2.5a) 
(2.5b) 



respectively analogous to the scattering length a and the 
effective range rp in s-wave scattering case. In the above 
equations, constants Ci^2 are determined by the details 
of the p-wave interaction at short scales, which in the 
pseudopotential model above are given by2i 



Cl 
C2 



ma 

2 2 
37r2 



A, 



(2.6a) 
(2.6b) 



where A = 27r/(i is the inverse size of the closed-channel 
molecular bound state, on the order of the interatomic 
potential range. 

The p-wave resonance and bound-state energy are de- 
termined by the poles of fp{k). At low energies (where 
ik^ can be neglected) the energy of the pole is given by 



kp 

^ 2m 



1 



mv I kf) I 



(2.7) 



which is real and negative and thus is a bound-state en- 
ergy for w > (negative detuning) and a finite lifetime 
resonance for v <Q (positive detuning). 

In the above, for simplicity we have focused on a rota- 
tionally invariant FR interaction, with lj and a indepen- 
dent of the molecular component i. This is an approxi- 
mation for our system of interest, the ®^Rb-^^Rb mixture, 
where, indeed, the p-wave FR around B = 257.8 
is split into a doublet by approximately Ai? = 0.6 G, 
similar to the fermionic case of 40 ^21,23,25,86 ^ leave 
the more realistic, richer case for future studies. 
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The background (nonresonant) interaction density 

Hbg = T~La + 'Hm + T~Lam (2-8) 

is given by 

= E ^^i'^'+Ai2V'IV'^V'2^i, (2.9a) 



a=1.2 



= |(0t.^)2 + ||^.^|2^ 
'Ham = ^ gam1pl(i>^ ' 0V'<t , 

(7=1.2 



(2.9b) 
(2.9c) 



where couphng constants Act, A12, 51,2, 5am are related to 
the corresponding background s-wave scattering lengths 
(oi, 02, etc.) in a standard way and thus are fixed ex- 
perimentally through measurements on the gas in a di- 
lute limit'^. Correspondingly, we take these background 
s-wave couplings to be independent of the p-wave detun- 
ing, an approximation that we expect to be quantitatively 
valid in the narrow resonance and/or dilute limits con- 
sidered here. A miscibility of a two-component atomic 
gas requires^^ 



aia2 > 



(2.10) 



which may be problematic for the case of ^^Rb-*^Rb due 
to the negative background scattering length of ^^Rb. 

The molecular interaction couplings 171, 52 (set by the 
L — and L = 2 channels of p-wave molecule-molecule 
scattering), and gam can be derived from a combina- 
tion of s-wave atom-atom (Ao-) and p-wave FR (a) in- 
teractions. We present lowest order of this analysis in 
Sec lIVD"3l which shows that these parameters can, in 
principle, be tuned via a magnetic field through the p- 
wave FR detuning v. 

The above two-channel model [Eq. (|2.ip ] faithfully cap- 
tures the low-energy p-wave resonant and s-wave nonres- 
onant scattering phenomenology of the *^Rb-^^Rb p-wave 
Feshbach- resonant mixture'*'^. Its analysis at nonzero bal- 
anced atomic densities, which is our focus here, leads to 
the predictions summarized in the previous section. 



binding atomic and molecular lattice-hopping kinetic en- 
ergies, density-density interactions, and a lattice projec- 
tion of the p-wave Feshbach resonant coupling that in a 
single-band Wannier basis is given by 



n 



latti( 
FR„ 



h.c 



(2.11) 

where, for example, on a cubic lattice, 6a are lattice 
vectors. A related finite angular-momentum FR lattice 
model was proposed and studied in an interesting paper 
by KukloA*^, predicting a robust p-wave atomic conden- 
sate in an optical lattice. As usualSS, at low lattice filling 
this lattice model reproduces the phenomenology of the 
continuum model. As an additional qualitative feature, 
at commensurate lattice fillings we also expect it to ad- 
mit a rich variety of zero-temperature Mott insulating 
phases and quantum phase transitions from them to the 
superfiuid ground states exhibited by the continuum sys- 
tem studied here. We leave the detailed analysis of the 
lattice model to future studies. 



2. Coherence-state formulation of thermodynamics 



With the model defined by H [Eqs. (EH), (g^, and 
(|2.9[) ]. the thermodynamics as a function of the chemi- 
cal potential /z (or equivalently total atom density, n), 
detuning ly, and temperature T can be worked out in 
a standard way by computing the partition function 
Z = Tr[e~^-^] {P = l/ksT) and the corresponding free 
energy F — —kBTlnZ . The trace over quantum me- 
chanical many-body states can be conveniently reformu- 
lated in terms of an imaginary-time (r = it) functional 
integral over coherent-state atomic, tpaiTjr) (cr = 1,2), 
and molecular, 0(r, r), fields: 



Z= DiP*^Dij^D(f)*D(f)e' 



(2.12) 



where the imaginary-time action is given bysi 



1. Lattice model 

As discussed in the Introduction, based on the ex- 
perience for the s-wave case^Sr^^iSS, it is likely that a 
stable realization of the above continuum p-wave res- 
onant two-species bosonic model will require an intro- 
duction of an optical lattic o"^^'^^ . This leads to a two- 
component atomic Hubbard model, with standard tight- 



s' / dT dr 



dr I drC. 



The Lagrangian density is given by 



(2.13a) 
(2.13b) 
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2m 

|2l;, |2 



0* • {Or 
|2 



4m 

2\ |j,|2 



+ I (0* • [V'l(-*V)V'2 - V'2(-«V)Vl] + C.C.) . 



(2.14) 



Above (and throughout), the summation over a repeated 
index, as for a in the first term, is imphed. 

We note that closely related models also arise in com- 
pletely distinct physical contexts. These include quan- 
tum magnets that exhibit incommensurate spin-liquid 
states^^ and bosonic atoms in the presence of spin-orbit 
interactions^. 

The associated coherent-state action S is the basis of 
all of our analysis in subsequent sections for the com- 
putation of the phase diagram, the nature of the phases 
and excitations in each of the corresponding phases of a 
p-wave resonant Bose gas. 

III. PHASES AND THEIR SYMMETRIES 

Before turning to a microscopic analysis, it is instruc- 
tive to consider the nature of the expected phases, corre- 
sponding Goldstone modes and associated phase transi- 
tions based on the underlying symmetries and their spon- 
taneous breaking. 

The fully disordered symmetric state of our two- 
component Bose gas confined inside an isotropic and ho- 
mogeneousSi trap exhibits the U n{^) ®U \n{^) ® 0{'i) ® 
Tr ^ T symmetries. The first two U{1) groups are as- 
sociated with the total (whether in atomic or molecular 
form) atom number = iVi -|- A'2 -f 2N,n and the atom 
species number difference AA^ = Ai — A2 conservations. 
The 0(3) X Tr symmetries correspond to the Euclidean 
group of three-dimensional rotations and translations (in 
a trap- free case), and T is a symmetry of time reversal. 

Since our system is composed of bosonic atoms and 
molecules confined to a large trap2^, at sufficiently low 
temperature we expect it to be a superfluid that in 
three dimensions exhibits BEG, characterized by complex 
scalar atomic, ^o-, and/or 3-vector molecular, order 
parameters. Thus, in addition to the high-temperature 
normal (non-superfluid) state, where the above order 
parameters all vanish and the full symmetry U]\r{l) (S 
Uan{^) ^ 0(3) ®T ®Tr is manifest^'', at low tempera- 
ture we expect the system to exhibit three classes of SF 
phases, 

1. Atomic Superfiuid (ASF), 7^ and * = 0, 

2. Molecular Superfluid (MSF), = and * 7^ 0, 

3. Atomic Molecular Superfluid (AMSF), 7^ and 



that spontaneously break one or more of the above sym- 
metries. Although these phase classes resemble the pre- 
viously studied phases of an s-wave Feshbach-resonant 
system32r— , as is clear in the following discussion, there 
are important qualitative differences. 



A. Atomic superfluid phases, ASF 

At large positive detuning v it is clear that the 
molecules are gapped and all atoms are in the unpaired 
open channel. In this regime, the gapped molecules can 
be neglected (or integrated out) and the Hamiltonian 
(|2.ip reduces to that of two bosonic atom species, that 
can exhibit BEG characterized by \l/2 condensates. 
Such a two-component system is characterized by two 
types of phase-diagram topologies and has been exten- 
sively studied in the statistical physics communitjJ^"-i2^. 

For aia2 > 0^2 it admits three ASF phases, 

1. ASFi (*i 7^0,*2 = 0), 

2. ASF2 (*i =0,*2 7^0), 

3. ASF12 (^-1 7^0,^-2 7^0), 

with ASFi and ASF2 separated from ASF12 and the 
normal phases by continuous phase transitions driven 
by temperature and density, or atomic polarization (or 
equivalently the chemical potential imbalance) as illus- 
trated in a mean- field phase diagram (Fig. |6]). These 
phases clearly break C/i(l), ^72(1), or both of these sym- 
metries, respectively, and are therefore expected to ex- 
hibit conventional Bogoliubov modes corresponding to 
these ViV) symmetries. 

Alternatively, for a\a2 < 0^2, the ASF12 state is un- 
stable, with ASFi and ASF2 separated by a first-order 
transition and the associated phase separation visible in 
a trap. 

We emphasize that, in contrast to the s-wave FR 
bosonic system (where atomic condensation necessarily 
induces a molecular one, and therefore the ASF phase is 
not qualitatively distinct from the s-wave AMSF phase, 
being separated from it by a smooth crossover)^^'^^, for 
a p-wave FR, above k = atomic ASF condensates do 
not automatically induce a p-wave molecular condensate 
since for fc = the p-wave FR coupling vanishes. Thus, 
the ASF class of phases is qualitatively distinct from the 
AMSF class that we discuss below. 
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B. Molecular superfluid phases, MSF 

In the opposite limit of a large negative detuning, 
atoms are gapped, tightly bound into heteromolecules 
that at low temperature condense into a p-wave MSF. In 
this regime of atomic vacuum, the gas reduces to that of 
interacting p-wave molecules, a system quite clearly iso- 
morphic to that of the extensively studied F = 1 spinor 
condensate^"— i^"—, with the hyperfine spin F here re- 
placed by the orbital £ = I angular momentum of two 
constituent atoms. 

Like F — 1 spinor condensates, the p-wave MSF can 
exhibit two distinct phases depending on the sign of the 
renormalized interaction coupling g2 in Eq. (j2.9b|) . or 

equivalently the sign of the difference ag™-* — a'"^^ of the 
molecular L = and L — 2 channels s-wave scattering 
lengths^l. 

1. Ferromagnetic molecular superfluid, MSFfm 

For §2 > the ground state is the so-called "ferro- 
magnetic" molecular superfluid, MSFfm, characterized by 
an order parameter, $ = ± irh), with n, rh, 

£ = fixrh a real orthonormal triad, a real amplitude, 
and the state corresponding to £z = ±1 projection of the 
internal molecular orbital angular momentum along the (. 
axis. MSFfni spontaneously breaks the time reversal, the 
0(3) rotational and the global gauge symmetry f7Ar(l), 
the latter corresponding to a total atom number N con- 
servation. Inside MSFfm the low-energy order parameter 
manifold is that of the 0(3) = SU (2) /Z2 group, corre- 
sponding to orientations of the orthonormal triad n, m, £. 

As its hyperfine spinor-condensate cousin, MSFfm, ex- 
hibits two gapless Goldstone modes, one linear (c>c k) Bo- 
goliubov mode associating with the broken global gauge 
symmetry and another quadratic (cx fc^) corresponding 
to the ferromagnetic order, with associated spin waves^ 
reflecting the precessional FM dynamics. 

2. Polar molecular superfluid, MSFp 

Alternatively, for 52 < the ground state is the so- 
called "polar — molecular superfluid, MSFp, character- 
ized by a (collinear) order parameter, $ — $pe*'^n, with 
h a real unit vector, (p a (real) phase, and $p a (real) 
order-parameter amplitude, with the state corresponding 
to £z = projection of the internal molecular orbital an- 
gular momentum along n. MSFp clearly spontaneously 
breaks rotational symmetry by its choice of the £z — 
quantization axis n and the global gauge symmetry, cor- 
responding to a total atom number conservation. The 
low-energy order parameter manifold that characterizes 
MSFp is given by the coset space {U{1) (E) S2)/'^2, admit- 
ting half-integer "charge" vortices^^ akin to (but distinct 
from) the s-wave MSF^S,-^. 



As we demonstrate explicitly in Sec. |Vl based on sym- 
metry we expect the polar MSFp state to exhibit three 
gapless Bogoliubov-like modes. One corresponds to the 
breaking of the global atom number conservation and two 
are associated with the breaking of rotational 0(3) sym- 
metry^. 



C. Atomic-molecular superfluid phases, AMSF 

As detuning is increased from large negative values 
of the MSFp^fm phases, for intermediate v the gap to 
atomic excitations decreases, closing at a critical value of 
j^MSF-AMSF which, in addition, an atomic EEC takes 
place. General arguments show that this precedes the 
atomic condensation in the absence of FR coupling; that 
is, j.,^SF-AMSF(^) ^ ^MSF-AMSF(o)^ The features of this 

MSF- AMSF transition and the AMSF phase are derived 
from the fact that at these intermediate detuning, the 
atomic condensation necessarily takes place at a finite 
momentum k = Q, set by a balance of the p-wave FR 
hybridization and the atomic kinetic energies. 

We emphasize that in contrast to the s-wave Feshbach- 
resonant boson a^^i^" , for which an atomic condensate 
necessarily induces a molecular condensate, thereby eras- 
ing a qualitative distinction between the AMSF and 
ASF states, for the p-wave case, ASF and AMSF phases 
are qualitatively distincli^^. The latter is ensured by 
the momentum-dependent nature of the p-wave coupling 
that breaks spatial rotational invariance and vanishes for 
Q = 0. 

As with other crystalline states of matter— i^^iSE, the 
detailed nature of the resulting AMSF states depends on 
the symmetry of the crystalline order, set by the recip- 
rocal lattice vectors, Q„, at which condensation takes 
place. Determined by a detailed nature of interactions 
and fluctuations, typically the nature of crystalline order 
is challenging to determine generically. Here we focus on 
the collinear states, with a parallel set of Q,j = uQ, that 
in the present system can be generically shown to be 
energetically preferred in the AMSFp state. There are 
two possible classes of such collinear states, which are 
bosonic condensate analogs of the Fulde-Ferrell (FF)S 
and Larkin-Ovchinnikov (L0)2i states, extensively stud- 
ied in fermionic paired superconductors and superflu- 
ids22r^. The qualitative features of these classes of finite- 
momentum superfluids are well captured by two simplest 
representative states, one with a single Q and the other 
with a pair of ± Q condensate, which we respectively de- 
note as "vector" (AMSF") and "smectic" (AMSF"). With 
a choice of Q the AMSF"'* states both break spatial rota- 
tional symmetry. However, they are qualitatively distin- 
guished by the AMSF" also spontaneously breaking the 
time-reversal symmetry, while remaining homogeneous, 
and the AMSF* instead also breaking the translational 
symmetry along Q, while remaining symmetric under 
the time reversal. Because within a mean-field theory 
analysis it is the former, vector state that appears to be 
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favored, for simplicity we focus on the single Q AMSF 
states. 

The nature and symmetries of these AMSF states fur- 
thermore qualitatively depends on the parent MSF, with 
AMSFfm and AMSFp as two possibilities depending on 
the sign of the renormalized interaction coupling (72- In 
addition to the symmetries already broken in its MSF 
parent, by virtue of atomic condensation the AMSF state 
breaks the remaining [/AAr(l) global gauge symmetry as- 
sociated with the conservation of the difference in atom 
species number, AN. Other symmetries that it breaks 
depend on the detailed structure of the AMSFf^^p states. 



1. Polar atomic-molecular superfluid, AMSFp 

The AMSFp emerges from the MSFp. As we see in the 
next section, in the AMSFp the finite-momentum atomic 
condensate orders with Q along the molecular conden- 
sate field and therefore (as illustrated in FigH]) for 
a single Q the vector superfluid does not break any ad- 
ditional spatial symmetries. With the molecular quan- 
tization axis, # locked to the atomic condensate mo- 
mentum Q, on general symmetry grounds (simultane- 
ous rotations of # and Q is a zero-energy Goldstone 
mode), we expect and indeed find that (see Sec. IV C|) 
the superfiuid phase will be characterized by a smectio^S 
Goldstone-niode Hamiltonian. The AMSFp superfluid, 
in addition breaks translational symmetry along with 
low-energy fluctuations about this state described by a 
smectic phonon u and a superfluid phase ip Goldstone 
modes. 



2. Ferromagnetic atomic-molecular superfluid, AMSFfm 

In contrast, a finite- momentum atomic condensation 
from the MSFfm leads to the AMSFfm. In this state, a 
p-wave Feshbach resonant interaction leads to the ener- 
getic preference for a transverse orientation of the atomic 
condensate momentum Q to the molecular quantization 
axis, £ = n X rh. Consequently, as illustrated in Fig. [SJ 
the AMSFfm state breaks additional orientational sym- 
metry of the uniaxial molecular state in the plane trans- 
verse to the molecular quantization axis £. That is, the 
AMSFfm state is a biaxial nematic superfluid defined by 
Q and £ axes, with the superfluid phase described by 
a smectic^^ Goldstone-mode Hamiltonian akin to that 
of the FF stated. The latter form is enforced by the 
symmetry associated with a simultaneous reorientation 
of atomic momentum Q and molecular gauge transforma- 
tion. The biaxial AMSFf^^^ superfluid, in addition breaks 
translational symmetry along Q, with low-energy fluctua- 
tions about this state described by two Goldstone modes, 
which are a smectic phonon u and a superfluid phase ip. 



A i = h X iri 




n 

A 




FIG. 4: (Color online) Schematic of the AMSFp state. The 
thick arrow indicates the atomic condensate momentum Q 
and the h arrow denotes the quantization axis along which the 
projection of molecular internal orbital angular momentum 
vanishes. 



FIG. 5: (Color online) Schematic of the AMSFfm state. The 
thick arrow indicates the atomic condensate momentum Q, 
lying in the plane transverse to the quantization axis I, along 
which the projection of the molecular internal orbital angular 
momentum is = -1-1. 



IV. MEAN FIELD THEORY 

Our main goal in this paper is to establish the phase 
diagram and nature of phase transitions exhibited by the 
p-wave Feshbach-resonant two-component Bose gas. This 
requires a minimization of the free energy which, in the 
presence of interactions and fluctuations is a nontrivial 
function of a number of systems' physical parameters. 
However, outside the critical region, inside each phase 
where fluctuations are smalliS^^ we can approximate the 
Landau free-energy functional ^] by replacing the 

atomic and molecular coherent state flelds with the clas- 
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sical order parameters, ^'^(r), ^(r), that minimize the 
action S via the saddle-point method. In the simplest ap- 
proximation, the Landau free-energy functional F['^cr, ^] 
takes the form identical to H[ipcr, 4>\, 



+ Al2|*l|'|*2|' 



cr=l,2 

9am (1*1 



1*21^)1*1' 



+ - (** • [*l(-^V)*2 - *2(-«V)*i)] + c.c.) 

(4.1) 

with the effective couplings (/io-, /2m, Ao-, . . . ), which are 
functions of the microscopic parameters (/ia- , , Ao- , . . . ) 
in Eq. (|2.ip . encoding all the complexity of the fluctua- 
tions and interactions on short scales. Though nontriv- 
ial, these parameters are, in principle, derivable from the 
Hamiltonian. However, we are not concerned with this 
aspect of the problem. Instead, our goal is to capture the 
qualitative form of the phase diagram, taking fluctuations 
into account only when they qualitatively modify the na- 
ture of the phases and phase transitions. For simplicity 
of notation, we therefore neglect the distinction between 
the microscopic and effective couplings, dropping tildes. 



A. Order parameters 

We begin by introducing order parameters that in 
mean-field approximation completely characterize the 
states of the system. In contrast to a conventional (s- 
wave interacting) Bose gas, anticipating the energetics, 
we allow the atomic condensates 5'i(r) and Vl/2(r) to 
be complex periodic functions characterized by momenta 
Q„, with the simplest single Qi= Q form given by 



V'i(r) 
V'2(r) 
0(r) 



*i(r) - *l,Qe»'?•^ 
*2(r) = *2,-Qe-^'?-, 



(4.2a) 
(4.2b) 
(4.2c) 



where $ is a complex 3-vector order parameter charac- 
teristic of the i — 1 molecular condensate and the choice 
of ± Q momentum relation for the two atomic condensate 
fields is dictated by momentum conservation. 

More generally, the atomic condensate order parameter 
is given by 



*i(r) 
*2(r) 



E 

0„ 



(4.3a) 
(4.3b) 



However, as alluded to in the previous section, based on 
the energetics of the model, we expect that for most of the 



phase diagram a single Q„ = Q and double Q^^ = zLQ 
coUinear forms of the atomic order parameters are suf- 
ficient to capture the ground-state atomic condensates. 
The latter LO-likc form can equivalently, more simply be 
written as 



vI/,(r) = vI/^,Qe'«•'- + vI/,,_Qe-^«•^ 



(4.4) 



with *CT,±Qj aud Q to be determined by the minimiza- 
tion of the mean-field free energy. As we demonstrate in 
Appendix \X[ it is the single Q (FF-like) condensate that 
is preferred energetically in a mean-field approximation 
and is therefore the primary focus of the analysis pre- 
sented here. 

The molecular condensate complex order parameter $ 
can, in general, be decomposed in terms of orthonormal 
real 3-vectors u and V)2i 

* = u + 2v. (4.5) 

As we demonstrate explicitly shortly, in this represen- 
tation the two possible i = 1 MSFs, ferromagnetic and 
polar condensates are described by 

u _L V, "ferromagnetic", iz = condensate, (4.6a) 
u II V, "polar", £z = condensate, (4.6b) 

where for ferromagnetic state u = v and the polar state 
can obviously be equivalently characterized by a vanish- 
ing of one (but not both) of u and v. These two molecular 
condensate states are the bosonic analogs of the + ipy 
and Px p-wave paired superfluidsSiiSl. 

We next consider the Landau free energy as a function 
of these atomic and molecular order parameters and, by 
minimizing it for a range of experimentally tunable pa- 
rameters, compute the mean-field phase diagram for this 
p-wave resonant two-component Bose gas. 



B. Atomic Superfluid (ASF) 



As is clear from Eqs. (|2.3p and (|4.ip for large positive 
detuning, v, the molecular chemical potential fi„i < 
is negative, with molecules gapped and therefore the 
ground state is a molecular vacuum. We can thus safely 
integrate out the small Gaussian molecular excitations, 
leading to an effective atomic free energy. 



J2 + ^i*.i 



+Ai2i^'in*2p 



(4.7) 



with coefficients that are only slightly modified from their 
bare values in Eq. (|4.1I) . This functional is a special 
U{1) (E) U{1) form of a 0{N) (g) 0{M) model, first studied 
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many years ago by Fisher et al. and more recently in 
magnetic and many other contextsiSi^-iS^. This free en- 
ergy is clearly minimized by a spatially uniform atomic 
order parameter, ^'o-, giving 



(4.8a) 



(4.8b) 



as the ASF free-energy density. 



ASFi 




N 



-jJLl 



ASR 



FIG. 6: Mean-field phase diagram of a p-wave resonant two- 
component Bose gas for large positive detuning. Molecules are 
gapped, reducing the system to a conventional two-component 
Bose gas, for A1A2 > displaying three types of ASF phases. 

A minimization of /asf, leads to four states correspond- 
ing to condensed and normal (nonsuperfiuid) combina- 
tions of the two-component Bose gas. For both negative 
chemical potentials, ^1 < 0, /i2 < 0, both atoms are in 
the noncondensed, normal (N) phase 



1*1 1 = 1*2! = 0. 



(4.9) 



On a lattice (e.g., generated by a periodic optical poten- 
tialiSi) at commensurate atom filling, this would corre- 
spond to a Mott insulating phase extending down to zero 
temperature. In a continuum (e.g., a trap), the normal 
state can only be realized by heating the gas above its 
degeneracy temperature. 

As physical parameters are varied (e.g., a weaker peri- 
odic potential, lower temperature, and higher density for 
one of the atomic species) for asymmetric mixture (dif- 
ferent densities and/or masses), one of the two atomic 
chemical potentials, fii,fJ-2 can turn positive, leading to 
a conventional normal-superfluid transition to ASFi or 
ASF2 states, respectively. The order parameters and 
mean-field phase boundaries in each of these conventional 



single-component atomic BECs are given by 



ASFi: ^-1 = /^,*2 = 0,forMi >0, < ^Mi, 
V '^i ^1 



ASF2: *i = 0, *2 = \f^, for ^i2 > 0, /ii < ^Ai2 

V A2 A2 



(4.10a) 

^12 

(4.10b) 



We note that generically for a symmetric two-component 
Bose mixture, these phases will be avoided by symmetry. 

Further changes in the system's parameters, so as to 
drive both chemical potentials positive, for A1A2 > A^2 
leads to ASFi - ASF12 or ASF2 - ASF12 transitions. The 
resulting two-component condensate, ASF12, is charac- 
terized by two nonzero atomic condensates and mean- 
field phase boundaries given by 



ASF12 



*i = 



A2/il — Ai2/i2 
A1A2 — X12 



,*2 = 



Ai/i2 — Ai2/il 
A1A2 — A?2 



for pLi > 0,fi2 > 0, 



> n > 

Ml 



A12 



(4.11) 



These classical phase transitions are generically continu- 
ous, in the XY universality class, breaking the associated 
C/(l) symmetries. The N-ASF12 transition only takes 
place in a fine-tuned balanced mixture /ii — ^2 (which is 
our primarily focus here) going directly through a tetra- 
critical point, fJ.i — fJ.2 — 0. Extensive studies demon- 
strate it to be in the decoupled universality classiSi^-iS^. 

For A1A2 < A12, the ASFi and ASF2 energies cross be- 
fore either becomes locally unstable. Consequently, in- 
stead of continuous transitions to the ASF12 state, the 
two-component ASF12 is absent and the ASFi and ASF2 
phases are separated by a first-order transition, located 
at 



Ml 



(4.12) 



which terminates at a bicritical point. On this critical line 
the ASFi and ASF2 states coexist and spatially phase 
separate. 



C. Molecular Superfluid (MSF) 

In the opposite limit of large negative detuning, that 
is, —v 3> open-channel atoms are gapped and the 
ground state is an atomic vacuum. Hence, for /i < the 
free energy F[^a-i ^] is minimized by ^'^ = and a uni- 
form molecular condensate The free-energy density 
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then reduces to 

(4.13a) 

/ 2 I 2\ I 5l / 2 I 2\2 I .92 / 2 2\2 
= -l-im[U +V ) + —(u +V ) +y(^i ) : 

(4.13b) 

identical to a spinor-1 bosonic condensate, corresponding 
to the £ = 1 molecular Bose gas. Thus, the thermody- 
namics and low-energy excitations of the MSF are iso- 
morphic to that of the well-studied spin-1 Bose-Einstein 
condensate^i^. 

The minimization of /nist[*] then leads to two super- 
fluid phases, the MSFp for 52 < and the MSFf^ for 
(72 > molecular condensates. For the polar MSF, the 
order parameter is given by 

* = . / n = $pn, for 52 < 0, (4.14) 

V 91 + 92 

spanning the [U{1) x S2]/'^2 manifold of degenerate 
ground states. For the ferromagnetic MSF, we instead 
find 

* = /pi(n+im) = ^(n+irh), for 32 > 0, (4.15) 
V 251 V2 

spanning the 50(3) manifold of states. In the above 
equation, n, rh, i = n x rh is an orthonormal triad and 
$p,fm are complex order-parameter amplitudes, breaking 



the 5*0(3) X ?7jv(l) symmetry of the disordered phase. 
For finite T the N-MSF transitions are in the well-studied 
universality class of a complex 0(3) model^. The MSFp 
and MSFfni are separated by a first-order transition, at 
f/2 = in mean-field approximation. 



D. Atomic Molecular Superfluid (AMSF) 

For the intermediate detuning, we consider a condensa- 
tion of both atoms and molecules, for generality allowing 
atoms to condense at a nonzero momentum. The latter 
is motivated by the discussion in the Introduction of the 
p-wave atom-molecule Feshbach coupling, which drives 
such finite momentum atom condensational^. 

To analyze the phase boundaries and the behavior of 
the order parameters in the AMSF phase, it is conve- 
nient to approach the AMSF state from the MSF phase 
at negative detuning, where molecular condensate is well 
formed, and study the atomic condensation upon the in- 
crease of the detuning and of the atomic chemical poten- 
tial. 

We focus on the simpler case of a single momentum, 
Q atomic condensate, that we also later find to be the 
preferred form of the AMSF state. We relegate to Ap- 
pendix |X] the conceptually straightforward, but techni- 
cally slightly involved, analysis of the more general ± Q 
momenta state. 

Using the order parameter form from 
Eqs. (|42a|) . (|4.2bp . and (j4?2cl) inside the mean-field 
free-energy density /amsf = ^]/V ^ fq + /msf , we 

obtain 
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*2,-Q*2,-q) - ^Q'^Iq%,-Q - A*Q^'i,Q*2,-^Q 



^'2,-q|'' + Ai2|*1,qP|*2,-qP, 



(4.16) 



where eg = §^ ^ fi + gam\^\^ Ag = a* • Q = |AQ|e»'^«, 
and for simplicity we specialized to a balanced mixture 
set by /xi = /U2 = M- To determine the nature of the 
atomic condensate in the AMSF state, we diagonalize 
the quadratic part of the free-energy density, fg with a 
unitary transformation Uq, 



obtaining 



(4.18a) 



Q 

(4.18b) 
(4.18c) 



where 



"Til 1 1 



(4.17) 



(4.19) 
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and 



= £Q-\^q\ 



Expressing the quartic terms of the free-energy density in 
terms of the diagonahzed atomic condensate fields, 'i>±, 
(4.20) we find 



J 



I* 



2 -Ql 



|*l,Qn*2.-Q|^ = 



4 ^' 



-2|*+p(^'+^'_ 



2|*+|2(*+^'_ + + 2\^-f{^+^- + *;*!) ] 



1^ + 



r 



(4.21a) 

(4.21b) 
(4.21c) 



Since < e^, the MSF-AMSF transition takes place 
0, tuned to this point by the FR detuning, 



at e 



Q 



V ^ V, 



MSF-AMSF 



At higher detuning, v > j^msf-amsf^ 
a finite momentum Q atomic condensate develops, char- 
acterized by a nonzero order parameter ^ 0, and 
\E'-i_ = 0. From the latter condition, we deduce that 



Q 



and 



= \/2e-''^°*i,Q, 



(4.22) 



(4.23) 



leading to a considerable simplification of the AMSF Lan- 
dau free-energy density. 



amsf 



1 



-M„|*|2 + |L|$*.$|2 + ^|$.$|2^ (4.24) 

where A = ■|(Ai + A2 -|- 2A12). The minimization of 
/amsf [^'-, over the order parameters and the atomic 
momentum Q is straightforward. The optimum |Qol = 
Qo is given by 



a/a: 



dQ 



0, 



and leads to 



Qo = am [(-u^ - v^) cos^ 6q + v^] 



21 1/2 



(4.25) 



(4.26) 



with Oq the angle between Qq and u. Without loss of 
generality, taking u > v and putting Qo back into the 
free energy shows that /amst is minimized by 6q = 0, 
that is, by Qq aligned along the longest of the u and v 
components, giving 



Qo 



amu w am^n„ 



(4.27) 



Thus, as illustrated in Fig. [51 the momentum Qo is at 
its maximum value near the MSF-AMSF phase bound- 
ary and decreases continuously to zero with the molecu- 
lar condensate at the AMSF- ASF transition, tunable 
with a magnetic field via detuning, v. 

As in the treatment of the MSF phases, it is convenient 
to express the free energy in terms of the magnitudes of 
the real, u, and imaginary, v, vector components of 
Minimizing it over we obtain 



2A 



^im{u +v ) + —[u +v ) +y(w ~v 



(4.28) 



with the atomic condensate given by 



ma 



9araiu^ + W^))/A 



1/2 



(4.29) 



Minimization of /amsf with respect to u and v gives a 
number of solutions. In addition to the normal (^f^ ~ 0, 
$ = u = w = 0) and the ASF 0, ^ = u ^ v = 0) 

phases, we find the AMSFp (*_ 7^ 0, it 7^ 0, w = 0) 
and the AMSFfm (*_ ^ 0, u > v ^ 0) phases that 
are the descendants of the MSFp and MSFfm molecular 
condensates. 



1. Polar AMSF: AMSFp 

A straightforward minimization of /amsf[w, w], 
Ea. K^ for 52 < leads to the AMSFp phase, 
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MSF, 



FIG. 7: (Color online) Schematic atomic (thick) and molec- 
ular (thin) order parameters versus the FR detuning v for 

MSFp-AMSFp , 

witn Vc\ = Vc and = 



the polar phase, 

AMSFn-ASF 



characterized by order parameters, 



K9\ + 52) - ~9. 



7,1 ' 

am 



(31+52)/^ - 



+52) -5, 



n2 

am 



(4.30a) 
(4.30b) 



where gam — dam — mo? jl. The phase boundaries corre- 
sponding to the MSFp - AMSFp and AMSFp - ASF tran- 
sitions are also easily worked out (set by the vanishing of 
the atomic and molecular condensates, respectively) and 
are given by 



,MSF„-AMSF„ 



,AMSF„-ASF 



52 - 25 am ) ^m i 
(.91 +92- '2gam) n, 



-{91 
1 

(2A - gam) ria 
(2A - gam) n, 



(4.31a) 

(4.31b) 

(4.31c) 
(4.31d) 



where we used — 2^ — v = [gi + g2)nm and /i = An^ 
to eliminate the molecular and atomic chemical poten- 
tials in favor of the molecular condensate Um , the atomic 
condensate Ua, and the detuning v. We also used the 
fact that at low temperature and for weak interactions, 
rim ~ n/2 and Ua « n in the MSF and ASF, respectively. 

It is clear from Fig.|H](a) and Eq. (|4.30bp for *_,p that 
the condition 



H91 + 92) - glm > 



(4.32) 



is necessary for the stability of AMSFp. We observe 
that in addition to setting the value of the finite mo- 
mentum, Qq of the atomic condensate, the p-wave FR 
coupling, a, expands the stability of the AMSF phase. 
Within the mean-field approximation, the MSFp-AMSFp 



and AMSFp-ASF transitions are of second order. 



This 



will be qualitatively modified, as we see when we discuss 
fluctuation effects in SeclVIl 




$ = 0,^_ 7^ 
ASF 




FIG. 8: Mean-field phase diagrams for polar phase as a func- 
tion of atomic and molecular chemical potentials, fia, fJ,m,, 
respectively. The ferromagnetic phase is similar but with dif- 
ferent parameters, (a) For X{gi + (72) — gam > 0, all three 
superfiuid phases — ASF, AMSF, and MSF — appear and are 
separated by continuous phase transitions (thick black lines), 
(b) For X{gi + 52) - glm < 0, AMSF is unstable, and the ASF 
and MSF are separated by a first-order transition (hatched 
double line). 



For A(gi -I- 52) - glm < [Fig. E (b)] , the AMSFp state 
is unstable, replaced by a direct first-order ASF-MSFp 
transition. The corresponding phase boundary is given 
by the degeneracy condition of the ASF and MSFp free 
energies 



/asf 



El 

'2A 



2(<?i + .g2) 



msfr> 



(4.33) 



2. Ferromagnetic AMSF: AMSFfm 

A minimization of the free energy, /amsf ["i*! for a 
range of couplings shows that for intermediate detuning, 
the low-temperature state is the ferromagnetic AMSFfm, 
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characterized by 



= y J\ -A ^ 1 \ ^7 ' (4.34a) 







(gi +92)9arn^J. + igi 


g2)gam^ ^" gamgamP^ra 




4:Xgig2 ' 


" ^g2gamgam - (.9l + 


.92)(ma2/2)2 






(.91 + g2)gamiJi + {gi 


~ g2)gamlJ' + gamgaml^m 






- ^g2gamgam. - (.9l + 


.92)(ma2/2)2 



'"fm^ \ TT— 7— — ~ / , „ 2 /o^2 ' (4.34b) 



1^ I ^ / .92(4.giA^ - 'igam^^m + ma'^fln,) 

' V 4A.9ig2 - 4g2ga™.9a™ - (51 + g2){may2)^ ' ^ ' ^ 




MSF 



The behavior of these order parameters as a function 
of detuning, v, is illustrated in Fig. [9] With increas- 
ing detuning, the component v (being smaller than u) 
vanishes first, signaling a transition of the ferromagnetic 
AMSFfni to the polar AMSFp state. Depending on the 
value of other parameters, upon further increase of v the 
system either continuously transitions at i/^^^^"^ to 
one of the three ASF states or undergoes a first-order 
AMSFfin-ASF transition with u discontinuously jumping 
to zero when v vanishes. As we discuss in SeclVl on gen- 
eral grounds, beyond the mean-field approximation, we 
expect the transitions from such smectic like superfluid 
phases (AMSFp^fm) to homogeneous and isotropic ASF 
states to be driven first-order by fluctuations. 



FIG. 9: (Color online) Schematic atomic (thick) and molecu- 
lar (thin and dashed) order parameters versus the FR detun- 
ing 1/ for ferromagnetic phases. The AMSFfm- AMSFp phase 
transition at t'c2 leads to kinks (change in slope) in the molec- 
ular (ti) and atomic ('I') order parameter, later indicated by 
a black dot. Without loss of generality we choose the n axis 
(component of u) to lie along Qg. The critical detunings are 
denoted by Uci — , i'c2 

, AMSFn-ASF 

and Uc3 = 



AMSFf„,-AMSFp 



The detuning phase boundaries corresponding to the 
MSFfm - AMSFfm and the AMSFfm - AMSFp transitions, 
determined by a vanishing of the atomic and the v (trans- 
verse to Qq) component of the molecular condensates, 
respectively, are given by 



^MSF,„,-AMSF,„ = _ (^^ _ 25„„ + maV2) (4.35a) 

« - i (.91 - 2g„™ + n, (4.35b) 

AMSFf„-AMSFp _ 8Aff2 + gam {^ma^ - 4g2) ^ ma^ (gi - 32 + ma^) 

4^2 + 2ma^ 



As with the polar state, the stability of the AMSFfm 
is dictated by a condition on the interaction couplings, 
given by 

4A5ig2 - 4g2gam5am - (51 + gi){jno? 12)'^ > 0. (4.36) 



In the opposite regime of 4A.91.g2 - Ag2gamgara - (51 + 
g2){ma^/2f < [Fig. [H] (b)], the AMSFfm state is un- 
stable, replaced by a direct first-order MSFfm-ASF tran- 
sition. The corresponding phase boundary is given by 
the degeneracy condition of the ASF and MSFfm free en- 
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MSF, 




Vc2 AMSFp Vc3 ASF 



FIG. 10: (Color online) Schematic detuning dependence of 
the momentum Qq of the atomic condensate starting with the 

TiHOT^ -4-1, MSFf„-AMSFf„ AMSFt„ -AMSFp 

MSFfm, With Ucl = Vc Vc2 = i^c , 

, AMSF„-ASF 

and Uc3 = 



ergies, 



7 asf 



2A 



91 



(4.37) 



3. Renormahzed molecular interactions couplings 

We conclude this section by noting that near a FR 
the microscopic pseudopotentials gi,Xi are modified by 
quantum fluctuations, replaced by corresponding experi- 
mentally determined scattering lengths. To lowest order 
(Born approximation, valid at low densities) in the FR 
coupling a, the diagrammatic corrections illustrated in 
Figs. [11] and Fig. [TH are given by 




FIG. 11: A lowest-order diagrammatic correction to molecular 
interaction coupling Qi. 



5gi 



15toA 



15mA 



97r 



ma 



(0.468) 



(4.38a) 
(0.0489)^, (4.38b) 



ma 



where a^g = ai -f a2 + 2ai2, the scattering lengths are 
defined by a standard relation, 



, and 

A sa 27r/(i is the ultraviolet cutoff set by the interatomic 





FIG. 12: Next-lowest-order diagrammatic corrections to 
molecular interaction couplings Qi. 



potential range. In the large A limit, Sg^ reduce to 



Sg^ 



Sg^ 



(0.468) 
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(2^) 



-(0.0489). 



(4.39a) 
(4.39b) 



This two-loop approximation (though valid only in the 
narrow FR limit), which finds 8g^ < 0, suggests that in 
the broad-resonance limit it is the MSFp that prevails. 

More generally, the importance of these fluctuation 
corrections to molecular interactions is that they pro- 
vide a mechanism to tune and, in principle, even change 
the sign of the effective g2, thereby allowing a detuning- 
driven MSFp-MSFf^ transition. 



V. ELEMENTARY EXCITATIONS 

Having established the existence of a variety of super- 
fluid ground states, we now turn our attention to the 
nature of low-energy excitations in each of these phases. 
As long as fluctuations remain finite for a range of a sys- 
tem's parameters, the phases detailed in the previous sec- 
tion are self-consistently guaranteed to be stable in these 
regimes and to retain their qualitative form. 

We study quantum fluctuations within each of the 
ASF, MSF and AMSF classes of phases established 
above. To this end we expand the atomic and molecu- 
lar bosonic operators around their mean-field condensate 
values Vl/cr, 



(pi = 



Stpa 



(5.1a) 
(5.1b) 



where 6ipa (f — 1,2) are fluctuation fields for atoms of 
flavors 1 and 2, respectively, and d<pi {i — x, y, z) are 
triplet of the £ = 1 molecular fluctuation fields. 

For some of the analysis it is convenient to work in 
momentum space, 

Sipa = ^^a^,fce**^ (5.2a) 

k 

5cf>, - ^E^'^'^^'"" (5.2b) 
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Using the above momentum representation inside the 
Hamihonian [Eq. (|2.ip ] and expanding to second order 



in the fluctuations operators Oo-.fc, we obtain H 

H„,ft[*^,*] with 



k 



Q!CT,fc • b^acr,fc+Q^ + /i.e. 

a 



(5.3a) 
(5.3b) 



r 



where /i^ is a Bogohubov Hamihonian matrix defined 
by matrix elements 



Ai2|vI/^P+g,„,|*|2, 
(5.4a) 



f^i.k 


= ^fk - Mm + .gi|*P + (51 - 


^252)|<i>.P, 




+ 5am(l*l|' + l*2n, 


(5.4b) 


A. 


2 ' 


(5.4c) 




= Ai2*i*;, 


(5.4d) 


^2,k 


= Ai2*^^'2 - a** ■ k, 


(5.4e) 






(5.4f) 




= 5i$*$j+2.g2$*$„ 


(5.4g) 


In 




(5.4h) 


OLa=(\,2).k 


= ±avE',,Q^(Q,-fc/2), 


(5.4i) 




= 2,2 = 1, and we suppressed the Q subscript 



on the atomic condensate order parameter, ^a,Q- The 
ten-dimensional bosonic Nambu spinor Ca.k is given by 



A diagonalization of this ten-dimensional Bogoliubov 
Hamihonian, preserving bosonic commutation relations 
of the Ca^k components gives the spectrum of the five 
modes throughout the phase diagram. This can be done 
numerically, but is not very enlightening. Instead, we 
study the problem one phase at a time, which allows a 
significantly more revealing solution of the problem. 
A. ASF phases 



(5.5) 



In the simplest limit of a large positive detuning, 
V > jyAMSF-ASF^ |-]-^g molecules are gapped, one or both 
species of the atoms are condensed at zero momentum, 
Q — 0, and the system is in the ASF phases. As discussed 
in Sec. Illli these are conventional well-studied superflu- 
ids, characterized by one Bogoliubov mode for each of 
the atomic U{1) symmetry that is broken. In the ASF 
phases $ = 0, the three molecular modes are gapped 
and can therefore be integrated out (adiabatically elim- 
inated). Away from the transition, this leads to only 
a small renormalization (that we will neglect) of effec- 
tive parameters in the resulting Hf. From Eq. (I5.3ap 
the atomic sector of the Bogoliubov Hamihonian is then 
given by 



H 



ASF„ 



E 



E 



k Lo-=l,2 



j,k + ^aaa.-k0.a,k ] + tlO-^ j.aw,k + t2,k0.a-kO'a,k 



h.c. 



(5.6) 



1. ASFc,: single atomic species BEC in an ASF^. phase. Standard analysis then leads to a 

conventional, gapless atomic Bogoliubov sound mode for 

In the regime where only a single atomic species of 
-01,2 condenses (i.e., '^a 7^ 0, = 0), the system is 
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species a 



(a) ^ 
ka 



2m \2m 



(5.7a) 
(5.7b) 



and a gapped atomic mode for the 



with Ca " 
complementary atomic species 

A:2 



2m*_ 



(5.8) 



where ^ ~ ni = 712 for a balanced case. Above, the 
coupling parameters are those from Eq. (|4.1I) . with # = 
= 0, and TOj- are effective atomic masses renormalized 
by interaction 



1 

m* 
1 

m* 



1 

771 
1 

m 



2(j/- An+ ^n)^ 
4(j/ - An + ^n) ' 



(5.9a) 
(5.9b) 



The remaining three molecular-like modes (corrected by 
coupling to atoms) are gapped and in a A; — >■ limit are 
given by 



(™) _ — 



kl 

(m) 
k3 



A, 



^ + (5.10a) 



k2 



Am 



^(7 , 9am 

-z-n H —n - fia 



(5.10b) 




kl,2 



FIG. 13: Schematic ASF single BEG (ASF<t) excitation spec- 
trum. The lowest curve is the atomic Bogoliubov mode and 
the upper curves are gapped atomic (thin) and molecular 
(thick) modes. 



L'^(l) symmetries spontaneously broken, then leads to two 
gapless atomic Bogoliubov sound modes for species 1 and 
2. Together with the gapped molecular excitations this 
leads to spectra of the five modes: 



^(112) 


\l 2mi 


2;^ + ^' 


(5.11a) 






771(112) 

Ek2 






(5.11b) 


p(™12) 

Eki 


ni(™i2) 
- Ek2 


= - 2Xn - 

Am 


V gam.n, (5.11c) 


p(mi2) 
^k3 




V - 2An -1- gam-n, 


(5.11d) 


- 1—: + 

Am* 



where for E, 



and E, 



(mi2) 
fc3 



we took fc — )■ and a — )■ 



limit and defined the sound velocity and effective atomic 
mass: 



^(ai2) _ / (A-Ai2)n ino? {\ - \i2)mn 
V m A(v — 2Xn + gamn) ' 

J_ _ J_ ^ 3(;/ - (A -)- Ai2)w -I- gaynn)na^ 
m* m (v — 2An + gamn)^ 



(5.12a) 
(5.12b) 



Ei::''^ and e[t^ are atomlike, gapless, in-phase and out- 



^k2 



of-phase modes, respectively. £'^2"'' ^I™^^' mod- 
ified by the FR interaction between atoms and molecules. 
The ASF-AMSF phase boundary is determined by the 
point where the molecular gap 



E. 



lASF 



= V — 2Xn 



(5.13) 



closes, and is consistent with the critical detuning deter- 
mined by the development of the molecular order param- 
eter that we found in Sec. ITVl 



'{inn) 




FIG. 14: Schematic ASF double BEG (ASF12) excitation 
spectrum. There are two gapless atomic Bogoliubov modes 
(thin) as well as three gapped molecular modes (thick). 



S. ASF12: double atomic species BEC 

In the regime where both atomic species of "01^2 con- 
dense, that is, 'J'i,2 7^ 0, the system is in a two-species 
ASF12 phase. Standard analysis, consistent with two 



B. MSF phases 



In the opposite limit of a large negative detuning, 



V < V 



MSF-AMSF 



both atomic species are gapped, 



0, 
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and p-wave molecules are condensed into one of the two 
(£ = 1)4=0 MSFp and £^ = ±1 MSFf„i, isomorphic to 
spinor-1 condensates with weh-studied properties^^i^i^. 
To see this, we note that the atomic Bogohubov exci- 
tations are gapped and can therefore be integrated out. 
Away from the transition, they lead to only a small renor- 
malization of effective parameters. Neglecting these small 
effects, the vanishing of ota-.k = ±a^cr,Q {Qa ~ V^) = 
decouples the Hamiltonian, Hf — Ha + Hm into atomic 
and molecular parts, that then are straightforwardly di- 
agonalized. 

The atomic sector. Ha is of standard Bogoliubov form, 
simplified to a 2 x 2 form by = Ao- = inside the MSF 
phases, leading to the atomic excitation spectrum, that 
for the symmetric case of /ii = = M is given by 



from the value of detuning at which the finite-momentum 
atomic order-parameter became nonzero. 

The diagonalization of molecular part H„i is also 
straightforward, and is identical to the case of the spinor- 
1 condensatea^iSi^, with effective parameters of our 
physically distinct, p-wave resonant scalar Bose gas. Sub- 
stituting characteristics of the polar phase MSFp (order 
parameters, /x, ~ (<?i + ^?2)'^m, ff2 < 0, etc. from 
above) into we obtain 



H 



MSFp 



k L 



Qefc+ (.91 +32)rtm^ ^|,fe^lK* 



Eff = V(£fe+|a*-fc|)(efe-|a*-fc|), (5.14) 



2 (.91 + g2)nmb\\,-k^,k 



where Sk = k"^ /2m — + .gam|*P- 

One key observation is that already inside the MSF 
phases the atomic spectrum, E^^^ (degenerate for cr = 
1,2 species) develops a minimum at a nonzero momen- 
tum femin — Qp^fmi with the corresponding atomic gap 

minimum, E^gap'^™, given by a value dependent on the 
nature of the MSFp ji„ phase. 



1. MSFp state / 52 < 

As analyzed in Sec. llVi the MSFp phase is defined by 
a molecular condensate order parameter, which can be 
taken to be a three-dimensional real vector, $ = tt = 
$pn, with Um = |$pp. In terms of the molecular con- 
densate density rim ~ n/2 the atomic chemical potential 
for the symmetric case, /ii = /i2 = is given by 

^J■ = ^(/"m + J^) = ^ ((51 + 92)nm + v) , (5.15) 

controlled by the FR detuning, v. 

For this symmetric case fJ-i = fJ-2 = fJ- (easily general- 
izable for the asymmetric, imbalanced case), the atomic 
spectrum minimum is characterized by 




£;(MSFp,a) _ , „ „ 



(5.16a) 
(5.16b) 



-g2nrah±-k ■ b_L,fc + h.c. 



(5.18) 



where bj^^fc are two degenerate transverse (to Qp) molec- 
ular modes. This leads to three Bogoliubov-type disper- 
sions, 



E, 



MSFp 
,fe 



4(<7i 



92)nmek, 



(51 + 92)nr, 



2m 



E 



MSFp 
_L,fe 



2 ^% + M92\nmek, 



\92\n.„ 
2m 



(5.19a) 

(5.19b) 
(5.19c) 

(5.19d) 



MSF 

where the longitudinal mode, E^^ ^ ^ describes the con- 
ventional MSF phase fluctuations and the doubly degen- 
erate transverse mode, E^^j^^ is the dispersion for the 
£ = I molecular orientational spin-waves. From the sec- 
ond set of A: — expressions we read off the correspond- 
ing phase and spin-wave velocities, given by 



MSFp 



MSFp 



(51 



92)nr. 



2m 



\92\n„ 
2m 



(5.20a) 
(5.20b) 



2. MSFfrn state : 92 > 



where in an isotropic trap the orientation of fcmin is spon- 
taneously chosen. The MSFp-AMSFp phase transition 
boundary is set by the closing of this atomic gap and is 
given by 



,MSFp-AMSFp 



(51 + .92 - 2.g 

am + ma'^)nm- (5.17) 



Reassuringly, this is identical to the critical detuning 
for this phase boundary, which we obtained in Sec. IIVI 



Inside the MSFfm state, the molecular condensate or- 
der parameter is given by $ = ^^('^ ~^ irh), expressed 

in terms of an orthonormal triad, n x rh = £. From 
the earlier mean-field analysis, the molecular condensate 
density is given by n„i — = /^m/ffi, leading for the 
symmetric case, fii — IJ-2 = fJ- 

^J' ^ 7; (91^771 + 1^) ■ (5-21) 
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FIG. 15: Schematic excitation spectrum for the MSFp. The 
doubly degenerate atomic spectrum (upper thin curve) ex- 
hibits a minimum gap at nonzero k, a precursor of finite- 
momentum atomic condensation inside the AMSFp. The 
molecular spectra (thick curves), one longitudinal (lowest) 
and two degenerate transverse (middle) modes, are of Bo- 
goliubov type. 



To lowest order, the atomic spectrum inside MSFfm has 
identical structure as that of the MSFp state [Eq. (|5.19p ]. 



but with the replacement gi + g2 ^ gi and Q!^/2, 

1 , , , 

— --j=amy'n.m, (5.22a) 

^(MSF,„,a) ^ g^^^^^^ a^^^ (5_22b) 

The MSFfin-AMSFfni phase transition boundary is deter- 
mined by the vanishing of the atomic gap, and is given 
by 

^MSF,„-AMSF.„, = _ j^g^ _ + ima'^ n^, (5.23) 

identical to the critical detuning obtained from mean- 
field theory for the order parameter in Sec. IIVI 

Using the above parameters characteristic of the 
MSFfm phase inside the molecular sector of the 

Hamiltonian reduces to 



H 



MSFf, 



k 



[4 



(5.24) 



r 



where 



1 

1 



(6„ -I- ibyn), 

{bn ~ ibrn), 



(5.25a) 
(5.25b) 



are expressed in terms of operators b„, bm, that are com- 
ponents of b along n, m, respectively. Diagonalization of 
the above Hamiltonian then gives the following spectrum 



MSFf„ 
k 

MSFf„ 
k 

MSFf„ 



1 

— ( 

2 
1 

2^* 



-ek 



4m' 
2g2n„ 



2V4 



linm<^k, 



girim 
2m 



(5.26a) 
(5.26b) 
(5.26c) 
(5.26d) 



where the Bogoliubov sound speed is given by CMSFt„, = 

We note that despite a three-dimensional coset space, 
S0(3) characterizing MSFf,„, only two modes (linear and 
quadratic in k) exhibit a spectrum that vanishes in fc — 



limit. The spectrum i^^^Ftm ^^g^^^ conventional 
Bogoliubov superfluid phase, here associated with the 
U{1) broken gauge symmetry of the molecular conden- 
sate. The quadratic in k gapless spectrum is that of 
the ferromagnetic spin waves, where the two components 
of the spinor are canonically conjugate and, as a result, 
combine into a single low-frequency mode. 



C. AMSF phases 

To obtain the spectrum inside the AMSF phases re- 
quires a solution of the fully general Hamiltonian, Hf 
[Eq. (|5.3ap ]. Because in this superfluid state all atomic 
and molecular modes are coupled, a direct BdG anal- 
ysis generically involves a diagonalization of a 10 x 
10 Bogoliubov matrix. This can be done numeri- 
cally. However, instead, below we take a complementary 
coherent-state path-integral approach that allows us to 
obtain the modes and dispersions analytically, leading 
to more insight into their structure. Using the formu- 
lation of the problem introduced in Sec. Ill 21 we ana- 
lyze the low-energy fluctuations in the AMSF states us- 
ing the coherent-state Lagrangian density, Clipa-Ttp] = 
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MSFr, 



FIG. 16: Schematic excitation spectrum for the MSFfm. The 
doubly degenerate atomic spectrum (thin curves) exhibits a 
minimum gap at nonzero k, a precursor of finite momen- 
tum atomic condensation. The molecular spectrum (thick 
curves) consists of a longitudinal gapless quadratic ferromag- 
netic spin-wave mode (lowest), a Bogoliubov sound mode, and 
a quadratic gapped mode. 



CuFTi'Ha,^] +SC [Eq. (EISl)], where £mft[*.,*] is 
the mean-field Lagrangian defining the AMSF phase and 
SC is the Lagrangian density of the quadratic fiuctua- 
tions. To obtain 6C we expand the atomic and molecular 
bosonic fields V'o-,'/' about their mean- field values (for 
clarity of notation in this section we choose to use p in- 
stead of n of the previous sections, where pa- = na/2, 
Pm = rim, and ps = n), 



= VPm'^e*'^ 



(5.27a) 
(5.27b) 



where Q„ = ±Q for cr = 1,2, respectively, Pm = 
Pmo + Spm and Per = pq + Spcr are the molecular and 
atomic densities, with the mean-field values pmO = 
and po = I^'ctP, and, based on Eq. (|4.22p . with the latter 
a independent in the AMSF phase. In addition to the 
density fiuctuations, 5pm,5pm and two atomic and one 
molecular superfluid phases, Oct, </3, the molecular Gold- 
stone modes are characterized by a unit vector, ^, whose 
form depends on the polar or ferromagnetic nature of the 
AMSF state: 



cj) = h, for AMSFp 



1 



V2 



{n + im), for AMSFfn 



(5.28a) 



(5.28b) 



Substituting these parametrizations of the atomic and 
molecular fields into the Lagrangian, Eq. (|2.14l) . we ob- 
tain 6£ that controls fluctuations in the AMSF phases. 



1. AMSF„ 



Focusing first on the polar state, with — y^p^ne*^, 
we find 



p.iidrO, - /v) + ^(V^?. + Q„f + Pm{idr^ - Mm) + T^CVy')' + 7^(Vn)^ 



Pn 

Am ^ 



oc^p^^^^wi n ■ {V9i - V6I2 + 2Q) cos((^ - 6*1 - 62) 

3„2 



1 



8to/3( 



Am 



+ A12/OI/O2 + 9am{pi + P2)Pm + -j^Pm " C,MFT[Pa, PmQ, '"•0, Q]i 



iSp+dr9+ + '^{Ve+f + iSp-drO- + ^(V0_ + Qf + i6pmdrip + ^{V^f + ^{Vnf 
m m Am Am 

2apoy/pmo n ■ (V0_ -I- Q) cos((/? - 26+) 



Po, 



PrnO , 



PmO , 



16mp( 



-(Vp+)' 



1 



16mp( 



■{Vp-f 



1 



A 



A 



A 



(Vp™)^ -I- '-5p\_ + '-5pt + '-^{5p\ - 6pt) + gamSp+Spm + "^ipin, 

(5.29b) 



(5.29a) 



5 v„2 



16mp,no 



where g = gi + g2, A = Ai = A2 for simplicity, and 







(5.30a) 


Sp± = 


Spi ± 5p2, 


(5.30b) 


p = 


^(Mi +M2), 


(5.30c) 


h = 


^(Mi -M2), 


(5.30d) 


Q = 


am^/p^fiQ. 


(5.30e) 



In the second form [Eq. (|5.29bp ]. we expanded the La- 
grangian about its mean-field value £mft to quadratic 
order in fiuctuations, 9„,Lp,5pa,5pm, and neglected the 
constant and subdominant contributions, that are negli- 
gible at long scales and low energies. We note that, as 
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usual, the linear terms in 5C-p [Eq. (j5.29bl) ] vanish identi- 
cally, enforced by the saddle-point equations for the con- 
densates, p_Oi Pmo, and Q. 

Examining the last form of (5£p, it is clear that im- 
portant simplifications take place at long scales. In par- 
ticular, the Feshbach resonant (Josephson-like) coupling, 
—acos,{tf — 26*+), between the closed-channel molecules 
and atoms (which is always relevant in three dimensions 



and therefore acts like a "mass") locks their phases to- 
gether at low energies giving 



if = 29^ 



(5.31) 



Integrating ip out and completing the square for the 
V6'_ -|- Q and n, to lowest order then gives 



J 



6Cu 



i{6p+ + 25pm)dr0+ + ^{Ve+ f + iSp^drO- + ^(V0_ + Q - am,/^fif + ^{Vhf 
m m 4m 



1 



16mpo 
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16mp( 
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A 



A 



A 



IGmpmo 



i(Sp+ + 2Sp,n)dr9+ + —(V0+Y + iSp^drO- + —{V9- - am^f^SnY 
m m 



4m 



IGmpo 



ii6p+ + 2Spm)dr9+ + ^{V9+Y + iSp^dr9- + '—{d,9^f + ——(VVi_9-f 
m 771 4m-^Q!^ 



1 



16mpo 



1 



16mpo 



A 



A 



A 



(Vp,„)" + '-5p\ + '-5pt + '-^{5p\- - 5pi) + gamSp+5p,^ + ^Sp^, 

(5.32c) 



16mp. 



Wo 



(Vp„)^ + '-5p\ + '-5pi + ^{Spi - Spi) + ga,nSp+Sp,n + ^Spl^, 

(5.32a) 



— ^(Vp_)2 + — i {Vpmf + ^Spl + j5p^_ + ^{Spl - b(?_) + ga^bp+bp^ + \bpl,, 

Ibmpo Ibmpmo 4 4 4 2 

(5.32b) 



g 



r 



where 



PsQ — Po + PmO, 



(5.33a) 
(5.33b) 



and in the second form [Eq. (j5.32b|) ] we used the mini- 
mum value of Q [Eq. (|5.30ep ] characterizing the AMSFp 
phase, which leads to a minimal-like coupling between 
V6'_ and 6n, the latter transverse (_L) to no and Q. Sub- 
sequently, to obtain our final expression, we integrated 
out Sn that to lowest order via a Higgs-like mechanism 
introduced a low-energy constraint 



Using it inside the (Vn)^ term then leads to a quantum 
smecticlike "elasticity" for the 0_ Goldstone mode, with 
z chosen to lie along Q, that is, z — Q. This smectic 
dispersion is expected based on the underlying rotational 
symmetry, which is spontaneously broken by the periodic 
AMSFp state. It is closely related to other periodic su- 
perfluids, such as, for example, the Fulde-Ferrell-Larkin- 
Ovchinnikov pair-density wave statea^^i^ii^i^. 



bfi 




(5.34a) 
(5.34b) 



As a final step we now integrate out the densities Sp± 
fluctuations, obtaining at long scales (where Vp± can 
be neglected) our final form for the Goldstone mode La- 
grangian in the AMSFp state: 



bjr, = l^^iO.9^f + ^{Ve^)' + lx-{dr0-)' + ^{d.9_r + -^{Vl9^ 
2 m 2 m Am-^a'^ 



(5.35) 



r 



where the compressibilities are given by 

2 

X- 

x+ 



A — Ai2 

g + 4g+ - 4ffa,n 
9+9 - 9lm 



(5.36a) 
(5.36b) 



with g+ = i(A + A12). 
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Thus, the in-phase and out-of-phase Goldstone modes 
are characterized by dispersions: 

aj_|_p(fc) ~ c+k, (5.37a) 

w^p{k) = ^{Bkl + Kk\)/x-. (5.37b) 

with defined parameters 



(5.38a) 



B 
K 




(5.38b) 
(5.38c) 



The linear uj+{k) dispersion of the superfluid phase 0+ 
is the expected Bogoliubov mode corresponding to the 
superfluid order. The anisotropic smecticlike dispersion 
of the "phonon" 0_ is a reflection of the uniaxial finite- 
momentum order in the AMSFp state, akin to the FF 
superconducto r T-^'^^ . 



FIG. 17: (Color online) The diagram defining various vectors 
appearing in Eq. (|5.39ap . V = ncos(<^— 2S+)— msin((/p— 20+), 
while tp — 261+ is measured relative to the ft axis and ipo is 
measured relative to z. 



X 




with only a single modification of the MSFfm order pa- 
rameter, given instead by (j) in Eq. (I5.28bp . The corre- 
sponding fluctuations Lagrangian density is given by 



m m 4m 



PmO 
71 

1 



1 



- Pmf — ^ctpo^/Pma ( + Q) ■[fi cos{ip — 26*+) — rh s\\i{Lp — 26^ 



16mpo 



(Vp+)= 



1 



IGmpo 



Wmpma 



i{Sp+ + 25pm)dr6+ + ^{Ve+f + iSp^drO- 
m 



PmO 

8m 
A 



(Vn)2 + ^(Vrh)^ 



16mpo 



Po_ 
m 
1 



4 

(Vp. 



(5.39a) 



T2 



ani^ PnioSfi I + ipmoSn ■ drrfi 



1 



IQmpmo 



i^Pm)' 



' ' c" 2 c 2 '^12 / f- 2 c 2 \ c" f 9^ r 2 

4 P+ + 1 + ^ °P-' + 9amSp+6p,n + yVm, 



(5.39b) 



r 



where to get the second form we performed a gauge trans- rh ■ drfi = dr^a into dr^ and to simplify the FR term, 
formation to absorb the n — rh planar rotations angle 
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as well as subsequently integrated out </?, completed the (n — m) component of Srh 
square into a minimal-like coupling for V0_, and chose 
Q = amy/ ^Y^fiQ, similar to the polar state analysis of 
the previous section. 

Integrating out Sfi, with the effective minimal-coupling 
constraint [Eq. (|5.34bp ] and the constraint on the in-plane at long scales we find 



n • 6m — —m ■ Sn, (5.40) 



8m 16mpo IGmpo IGmpmO 

+ l^P+ + l^P- + ^^^P+ " ^P-^ + 9amSp+5p^ + ^5pl, (5.41a) 



+ iKdy0-drJ+^J{Vjf, (5.41b) 



where we used [V((5rh)]^ = [V(n • Srh)]" 
introduced couplings 



[V{£ ■ (5rh)]2, inside the AMSFf^ state: 



v2pm0 



am 
1 

m^a"^ 
1 



2m^a^ 

PmO 

Am 



defined a real scalar field 

7 = £ • (5m, 



(5.42a) 
(5.42b) 
(5.42c) 
(5.42d) 

(5.43) 



for fluctuations of rh outside of the fi — rh plane, and 
chose axes 



X = m, 
y = 1. 



(5.44a) 
(5.44b) 



We note that the Goldstone-modes action [Eq. (|5.41l) ] 
exhibits a biaxial smectic energetics in the smectic 
phonon, in addition to the xy- model energetics of 
the superfluid phase, The biaxiality is expected and 
arises due to a smectic in-plane polar (p-wave) order, 
characterized by a spinor, </>/m, with the quantization 
axis, t. The finite angular momentum, (.z — il along 
I distinguishes AMSFfm from AMSFp and leads to an 
additional Goldstone mode 7. 

A straightforward diagonalization of the above La- 
grangian leads to dispersions for three Goldstone modes 



(5.45a) 



= J[Bk:i + k:^{K,kl + Kykl)]/x-, (5.45b) 



IJB[Bkl + k-^{K^kl + Kykl)] 



(5.45c) 



The anisotropic ijjj^_^{k) dispersion corresponds to the 
ferromagnetic spin waves in the plane of atomic conden- 
sate phase fronts ("smectic layers") of the p-wave atomic- 
molecular condensate, AMSFfm, reducing to the disper- 
sion of MSFfni in Eq. (|5.26p for a vanishing smectic order, 
with B = 0. 



VI. PHASE TRANSITIONS 

In this section, we study the quantum MSF - AMSF 
phase transitions beyond earlier mean-field approxi- 
mation, demonstrating that they are described by a 
d-|-l-dimensional quantum de Gennes (Abelian Higg's) 
modei^ akin to that for a normal-to-superconductor and 
nematic-to-smectic-A transitions. Based on the extensive 
work for these systemsiS^iiS^, in three (spatial) dimen- 
sions {d = 3) we predict that the effective gauge-field 
fiuctuations drive this transition first order. The deriva- 
tion is most transparent via a coherent-state Lagrangian, 
Eq. (11331), 
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y2 y2 ^ 

£ = - Ai<x)'(/'t7 + (t>* -{dr- ^ira) ' </> + -^\'4>a\'^ 

Zm 4m 2 

+ Xl2\i^l\'\H' + 9am (l^lP + |V'2P) l^/"!' + ' + f |0 ' c/'P 

+ |(</.*-[^lHV)^2-V'2HV)V^i]+C.C.), (6.1) 



working in polar representation similar to that of the pre- 
vious section. 

A. MSFp-AMSFp polar transition 

It is convenient to analyze the transition from the MSF 
side, where the atomic and molecular order parameters 
are given by 

i'a ^^a,Qy^'-\ Qa = ±Q, for a =1,2 (6.2a) 
=Vp^e''^^'''>h. (6.2b) 



Using these forms inside C [Eq. (|6.ip ] and for simplic- 
ity focusing on the balanced case with pL = jl„ = — 

garnPinO, WC obtain 



2 \ 1 

2 I L;, |2\ I |T-7„;, |2 



=^lQdri'l,Q + ^l-Qdri'2,-Q + ( " M ) (1^1,01 + 1^2,-^1') + — |V7^,,Q 



2m 



4m 



PmO - \2 I 5r 2 I 52 2 |r |2 , 



+ C.C. 

(6.3) 



where terms linear in fields vanish by virtue of the saddle- 
point equations. The contribution £msf is the mean-field 
part analyzed in Sec. lIVI and Ci-^t is the higher order term. 
Defining 

eg = ^ - A, (6.4a) 
2m 

Aq = aVPmo" • Q, (6.4b) 
and introducing atomic eigenfields 

V'+ = + i'l-q)^ (6-5a) 

V"- = ^(V'l.Q + ^-2 -q): (6-5b) 



a mean-field version of which was obtained in Sec. IIV[ 
the Lagrangian simplifies considerably to, 
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TpldTip- + ij+dTijjl + - — I (— « V + Q + am^pmon cos ip)il'+\ + - — | (-i V + Q am^/p^fi cos ip) -tp. 



2m 



1 



e+ - ( Q + amJpman cos ip)' 
2m 



2m' 

e- - TT- ( <3 - amJprnan cos (^)^ 
2m 



2(7 4m 2(72 4m 



msf 1 



(6.6) 



r 



where 



e± =eQ±|AQ| 



(6.7) 



and we completed the square in £p. It can be shown 
that near a critical point the sin contribution leads to 
an irrelevant quartic correction to and renormal- 

ization of (SyT/i-)^ stiffness. Furthermore, it is clear that 
the canonically conjugate field ip+ (it appears as a canon- 
ical momentum for the critical field ip- ) remains massive 
at the MSF-AMSF transition, defined by the vanishing 
of the coefficient of IV'-P term, consistent with Sec. IIVI 
Therefore, safely integrating out -0+ and making a choice 
Q = am^prnonQ that minimizes the energy, leads to 



2m 



1 

2^2 



+ + S(V^f, (6-8) 

with A = ^(Ai + A2 + 2A12), and we dropped the mean- 
field part and irrelevant interactions. 



Thus, as anticipated on symmetry grounds, the zero- 
temperature MSFp-AMSFp transition is indeed described 
by a quantum {{d+ l)-dimensional) de Gennes model (or 
equivalently the Ginzburg-Landau) Lagrangian^^, where 
the role of the nematic director (gauge-field) is played 
by the £z = quantization axis of the p-wave molecular 
condensate. 



B. MSFfm-AMSFfm ferromagnetic transition 

Using the field forms appropriate for the ferromagnetic 
case 

V'^ ='/'^Q„e*«" •^ Q, = ±Q, for (7=1,2 (6.9a) 
(/,=y^(n + *m), (6.9b) 

a very similar analysis leads to 



|2 I 1 ITT / |2 , ■ - o " I /'™0/_-n2 , PmO ~ n2 

+ 7;— ^V'- +tpmn-dTm+- — (Vn) + - — (Vm) 

2m 8m 8m 



+ a^^^ih - im) ■ (V';(-iV)V'+ - C(-«V)^A_) 
v2 



Q 



2m 



(0;(-zV)V^+ + 7^1(-ZV)V-) - r+dr^^- + ^+drr- + ^Sp^ + A„t + C^.sf , 

1 



-iV + Q + --j=am^p„Mh ] 



1 
2m 



(6.10a) 
2 



-iV + Q --j=am^prnon ] ip- 



+ ^(Vn)2 + t^iVrhf + a^^i^zm) ■ {r+H^}^- - V+(+*V)V/ ) + ^5pl + C,nt + A„sf, (6.10b) 
8m 8m ^/2 '2 



2m 



Q+ —^am^praon 



1^4 



2m 



Q~ —^amy/praon 



IV'-P + ipmn ■ drTh 



\2 PmO 



/ PmO 



■9 c„2 



r 



where to obtain the final form we rotated h and rh by of the polar case in the previous section, here it can be 
—ip and completed the square. Similarly to the treatment 
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shown that the linear {—irh) term only leads to irrel- 
evant quartic coupling and can therefore be neglected. 
Integrating out the noncritical conjugate field ■0+ gives 
the final Lagrangian form 



2m 



V2 



+ ipmn ■ drin 



PmO 

8^ 



(Vm)2 
(6.11) 



of the quantum de Gennes-Ginzburg-Landau form that 
controls the MSFfm-AMSFfm transition. In the above we 
dropped the mean-field part and irrelevant interactions. 
As anticipated by symmetry, it is distinguished from the 
polar case by the additional biaxial order whose fluctua- 
tions are characterized by rh. 



VII. TOPOLOGICAL DEFECTS 

Having established the nature of the ordered states, 
characterized by Landau order parameters, and the asso- 
ciated Goldstone modes, we now turn to a brief discussion 
of the corresponding topological defects. As usual, these 
singular excitations are crucial to a complete character- 
ization of the states and their disordering, particularly 
in the case of non- mean- field (e.g., partially disordered) 
states that are not uniquely characterized by a Landau 
order parameter. 



A. Defects in ASF 

As discussed in Sec lIVi the ASF^ states (with i = 
1,2,12) are characterized by two atomic condensate or- 
der parameters, tpa = \fPae^^° ■ Correspondingly, as in 
an ordinary superfluid, because Qa are compact phase 
fields {9cr and 9^ + 27r are physically identified), in ad- 
dition to their smooth Goldstone mode configurations, 
there are vortex topological excitations, corresponding 
to nonsingle- valued configurations of 9^(r). These are 
defined by two corresponding integer-valued closed line 
integrals enclosing a vortex line 



W9, = 2np, 



(7.1) 



In a differential form, the line defects are equivalently 
encoded as 

V X = , (7.2) 
with vortex line topological "charge" density given by 

m,(r) = 2^1]/ T^Ms^)5''{Y-v,{si))ds, , (7.3) 



where Si parametrizes the i'th vortex line (or loop), Yi{si) 
gives its positional conformation, ti{si) is the local unit 
tangent, and vortex "charges" are independent of Si, 
since the charge of a given line is constant along the de- 
fect. Furthermore, 



V • m(r) = 



(7.4) 



enforces the condition that vortex lines cannot end in the 
bulk of the sample; they must either form closed loops or 
extend entirely through the system. 

Thus, vortices in the single-component ASFg. states are 
characterized by a n„ integer, and in the two-component 
ASF12 the defects are specified by a pair of integers 
(pi,P2)- These are associated with the fundamental 
group TTi of the torus U{\)®U{1), that characterizes the 
low-energy manifold of Goldstone modes of the ASF12 
state. It is therefore closely related to other U{1) (E) U{1) 
systems, such as easy-plane spinor-1 condensateaiSl and 
two-gap superconductors, for example, MgB2^. 

As in conventional superfluids vortices appear in re- 
sponse to imposed rotation and proliferate with enhanced 
quantum and thermal fluctuations, providing a comple- 
mentary description of phase transitions out of the ASF^ 
states. 



B. Defects in MSF 

Because of its finite angular momentum, i — 1, struc- 
ture the defects in the MSF states are somewhat more 
complicated. However, relying on the aforementioned re- 
lation of the MSF to the well-explored spinor-1 conden- 
sates— '^>^'^>^'^, we inherit a clear characterization of 
defects in the two MSF phases. As discussed in Sec. HV] 
the polar MSFp and the ferromagnetic MSFfm states are 
respectively characterized by [5*2 x t/jv(l)]/Z2 (the mod 
out by Z2 corresponds to the identification of n — > — n 
with ip ~> (f + n) and 5*0(3) order-parameter (Goldstone 
mode) manifolds. The defects are characterized by the 
homotopy group of the corresponding manifolds. In the 
MSFfin case the SO(3)=S3/Z2 manifold also appears in 
the dipole-locked A phase of helium-3 with topological 
defects well understoodi2£. 

The nature of defects in the MSFp state was a sub- 
ject of some debate, until it was definitively resolved by 
Mukerjee, et al^. These are characterized by elements 
of the homotopy groups 7Tn{S2 x f7Ar(l)/Z2) = Z. The 
key new feature is the appearance of a composite defect 
that is a TT vortex and h texture where n — > —n, keep- 
ing the molecular order parameter single- valued at long 
scales. The consequences of this were discussed and ex- 
plored through Monte Carlo simulations by Mukerjee, et 
al.—, and is quite closely related to other realizations of 
composite half-integer defects^"— 1^>^. We expect the 
MSFp to exhibit similar phenomenology, which we do not 
explore further here. 
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Defects in AMSF 



As discussed in Sec. HVl in addition to the molecular 
condensate (p, the two AMSF states are characterized 
by a finite momentum two-component atomic condensate 
order parameter, with a nonzero amplitude 



Q 



2,-Q' 



(7.5) 



and a vanishing amplitude vj/^ — Q [Eq. (14.19^ ]. The lat- 
ter is consistent with the locking of the atomic condensate 
phase 0+ — \{9x+92) to a MSF phase imposed by 
the PR coupling [Eq. (|5.29p ]. It also locks the atomic 
condensate magnitudes to be equal, I^'i^q] = |^2,-q|- 

Using the phase representation, the atomic condensate 
order parameter reduces to 



J6- 



-iip/2 , 



(7.6) 



From this form it is clear that, as a conventional super- 
fluid, the AMSF admits 2tt vortices in 9- = \{9i- 6*2), 
and = 0, corresponding to a 27r "spin" vortex, 

0_(r) = 9i{v) = -92{v), (7.7a) 
= 6*, 27r "spin" vortex, (7.7b) 

with equal counterpropagating (atomic species 1 and 2) 
currents and a vanishing "charge" (atomic number) cur- 
rent. Above, is a polar coordinate angle. 

Another type of a defect is topologically equivalent to 
a 27r vortex in 0+(r), 

9+{v) - 01 (r) = 02 (r), (7.8a) 
= 9, 27r "charge"-vortex, (7.8b) 

with equal copropagating (atomic species 1 and 2) cur- 
rents and a vanishing "spin" current. However, as is clear 
from the Feshbach interaction form in Eq. (|5.29bp . 



5L 



FBR 



cos((p — 29+), 



(7.9) 



for vortex- free molecular order parameter (e.g. ~ 0), 
inside the AMSF phase the "charge" vortex 2n wind- 
ing and currents are confined to a domain wall whose 
thickness is set by the ratio of the superfluid stiffness 
and FR coupling a, on the order of l/Q. As a result 
of this current confinement the energy of such domain- 
wall scales linearly in two dimensions and as a surface 
in three dimensions. Consequently, such ±27r "charge" 
vortices are confined into neutral pairs inside the AMSF 
phase. However, in the presence of a molecular 47r vor- 
tex, with (p(r) = 20+ (r) = 20, no domain wall appears 
and conventional ±27r "charge" vortices can deconfine. 

Finally, as with other analogous physical systems^^i^, 
the product form of the atomic condensate order param- 
eter, VP- [Eq. (|7.5I) ]. admits composite defects with half- 
integer topological charge. These are characterized by a 
bound state of a 7r-"spin" and ±7r-"charge" vortices, with 
latter (as above) confined by FR interaction into a ±7r 



domain wall. The simplest (topologically faithful) real- 
ization of this is a vortex only in one (but not both) 
atomic species, 

0+(r)=0_(r) = i0i(r) = i0, 02(r) = 0, 

(+7r) — (+7r) vortex domain wall, (7.10a) 

9i(r) =0, 



?+(r) = -0_(r) = i02(r) = i 



(— tt) — (+7r) vortex domain wall. (7.10b) 

Again, in the presence of a ±27r molecular vortex, <p(r) — 
±0, the 7r-"spin", 7r-"charge" composite vortex, 0_(r) = 
±0+(r) = I no longer exhibits a domain wall, since 1^ — 
20_|_ = 2iip. It is therefore not confined inside the AMSF 
state. 

Clearly, out of the above six types of defects, the 2-k- 
"spin" vortex is least energetically costly, because it does 
not involve a "charge" domain wall in 0+ nor require an 
additional molecular vortex. On the other hand, it is the 
two half-integer vortex domain-wall defects that are the 
elementary ones. This therefore opens up a possibility 
of unconventional nonsuperfluid states in the two-species 
p-wave resonant Bose systems, driven by unbinding of 
composite topological defects, like the 27r-"spin" vortex. 
We leave the discussion of the resulting states to future 
work. 
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FIG. 18: (Color online) (a) 27r-"spin" vortex with equal coun- 
terpropagating atomic currents, ji, j2. (b) 27r- "charge" vortex 
with equal copropagating currents, confined to a domain wall 
(gray area) of width ^ ~ 1/Q, with atomic order parameter 
suppressed. In the presence of a molecular An vortex a do- 
main wall is no longer required, and the "charge" vortex is 
deconfined. 
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(b) 



FIG. 19: (Color online) tt — tt vortex in a = limit, (a) In 
6*1 — Q-i representation; black (gray) arrows indicate Q\ (^2). 
(b) In 6*+ — Q- representation, the pair of arrows indicate 6*+ 
and while the wavy line indicates a domain wall. For 
a 7^ 0, the FR coupling "squeezes" the 6'+ 7r-vortex textures 
into a domain wall of width C ~ VQ- the presence of an 
additional molecular 2-k vortex, the domain wall is absent and 
the composite defect is deconfined. 



VIII. LOCAL DENSITY APPROXIMATION 

Because the primary experimental application of our 
predictions is to degenerate atomic gases it is important 
to extend our analysis to include the trapping potential 
Yt(y:')i which in a typical experiment is well-approximated 
by a harmonic potential. A full analysis of the effect of 
the trap is beyond the scope of this paper, and here we 
limit our treatment to a LDA. 

Closely related to the WKB approximation^^, LDA 
amounts to the bulk system predictions, but with the 
chemical potential replaced by an effective local chem- 
ical potential — /i — V((r). The validity of the 
LDA relies on the smoothness of the trap potential, 
with the criterion that V((r) varies slowly on the scale 
of the longest physical length A in the problem, i.e., 
{X/Vt{r))dVt{r) /dr <^ 1. Its accuracy can be equiv- 
alently controlled by a ratio of the single-particle trap 
level spacing SE to the smallest characteristic energy Ec 
of the studied phenomenon (e.g, the chemical potential, 
condensation energy, etc.), by requiring SE/E^ <C 1. For 
our system the longest natural length scale is the period 
27r/Q, Eq. (jl.ip of the finite- momentum atomic conden- 
sate inside the AMSF state. Thus, away from the AMSF- 
ASF phase boundary, where Q vanishes (see Fig. [2]), we 
expect an LDA treatment of the effects of the trap to be 
trustworthy. 

A generalization of a resonant Bose-gas model 
[Eq. I|2.1I) ] to include a trap is straightforward, accounted 
for by the additional Hamiltonian density 

Htrap = ^ yt(r)^i^<, + 2V^(r)0t.^^ (8.1) 

a=l,2 

with % ^ % + "Htrap- In the above, for simplicity we 
specialized to an atomic species-independent trapping 
potential and approximated the closed-channel molecu- 
lar trapping potential by twice the atomic one, valid for 
the interaction range rg (typically less than 50 A) much 



smaller than the cloud size R (typically larger than a 
micron) . 

Henceforth, to be concrete, we shall focus on an 
isotropic harmonic trap (although this simplification can 
easily be relaxed) with 



Vt{v) 



1 



2 2 

m uj^ r , 



(8.2a) 
(8.2b) 



the latter expression defining the cloud size R. Within 
LDA, locally the system is taken to be well-approximated 
as uniform, but with a local chemical potential given by 



/i(r) 



1 



= A* 



2 2 

-muj+r , 



r2 



(8.3a) 
(8.3b) 



where is the true chemical potential (a Lagrange multi- 
plier) enforcing the total atom number N . The spatially 
varying species 1 and 2 chemical potentials are then given 
by: 



M2(r) 



^(r) 



h, 



(8.4a) 
(8.4b) 



with a uniform chemical potential difference h set by the 
atomic species imbalanc o^^i^^i^^ . 

Consequently, within LDA the system's energy den- 
sity is approximated by that of a uniform bulk system 
[Eq. ()4.ip ]. with the spatial dependence entering only 
through /i(r). The ground-state energy is then simply 
a volume integral of this energy density. Thus, the phase 
behavior of a uniform system as a function of the chem- 
ical potential, fi, translates into a spatial cloud profile 
through fj,{r), with the critical phase boundaries /Xc cor- 
responding to critical radii defined by /Zc = A*(rc, h)^^^. 
As predicted22iiii^ and observedi^""-ii^ in other systems, 
this leads to a shell-like cloud structure "imaging" of the 
bulk phase diagram as illustrated in Fig. [201 

Applying this LDA analysis to our system leads to 
a prediction of rich, magnetic-field, atom-number, and 
temperature-tunable shell structures in a p-wave reso- 
nant Bose gas, schematically illustrated in Fig. [HI For 
a range of atom number, detuning, and temperature ad- 
mitting the AMSF phase, we expect a cloud shell with 
an r-dependent atomic condensate wavevector q{r), given 
by 



q{r) = am^Jnm{r), 



.5a) 



1 - for TMSF < r < rASF, (8.5b) 

where tmsf [T, N, v) and tasf [T, N, v) are the bound- 
aries of the AMSF sheU. 
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FIG. 20: (Color online) A'^, v, T-dependent cuts through the 
bulk phase diagram with increasing radial position r through 
the atomic cloud. Stars indicate system's parameters (local 
chemical potentials pa, fJ-m) at the trap center. 



IX. SUMMARY AND CONCLUSIONS 

To summarize, we studied a degenerate gas of two- 
species bosonic atoms interacting through a p-wave FR, 
as realized, for example, in a ^^Rb-^^Rb mixture. We 
mapped out the corresponding phase diagram and ther- 
modynamic properties of the phases as a function of 
temperature, atom number, and FR detuning, and ana- 
lyzed the nature of corresponding phase transitions. We 
showed that at intermediate detuning such atomic quan- 
tum gas generically exhibits an AMSF state with atoms 
condensed at a finite tunable momentum Q{h') along a 
direction set by the angular momentum axis of the molec- 
ular condensate. This AMSF state undergoes quantum 
phase transitions described by a quantum de Gennes 
model into a p-wave (orbital spinor-1) MSF and into an 
s-wave ASF at large negative and positive detunings, re- 
spectively. A magnetic field can be used to tune the 
modulation wavevector of the AMSF between zero and 
a value set by interactions as well as to drive quantum 
phase transitions in this rich system. 
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FIG. 21: (Color online) Schematic illustration of the shell 
structure expected in the p-wave resonant atomic cloud, cor- 
responding to the phase diagram cuts in Fig. 1201 



the initial part of this work. 



Appendix A: Order parameter structure 

As discussed in the main text, the detailed nature of 
the AMSF states depends on the structure (the set of 
reciprocal lattice vectors, Q„) of the finite-momentum 
atomic order parameter. However, because ^ q depends 
on the details of the inter atomic interactions and fluc- 
tuations, to determine its form in general is a nontrivial 
problem, as exemplified by the FFLO problem and the 
conventional crystallization. However, as seen in Sec lIVi 
for the case of the AMSFp state, the problem simpli- 
fies considerably as the energy is clearly minimized by 
a collinear state, with Q„ parallel to Such collinear 
states fall into two universality classes®'', represented by 
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the FF-like-^ and the LO-hke''- single harmonic forms, 

*r = *.,Q„e^^^ ^ (Al) 
= vI/,,Qe^Q'^ + ^,,-Qe-'^r (A2) 

In the FF-hke (LO-Hke) state each species is character- 
ized by a single Q (double ±Q) momentum, exhibiting a 
uniform (periodic) atomic density. 

Focusing on these two coUinear (FF and LO) states, in 
this appendix we demonstrate that, generically (at least 
within the mean-field theory), it is the FF state that is 
energetically selected by the interactions. 

To this end, we reexpress the mean-field energy den- 
sities for FF and LO in terms of the corresponding 
eigenmodes, the latter involving two (iQ) critical 

modes, 

£^^^{sQ^\AQmQf + h\^^\\ (A3) 
£:LO-(eo-|AQ|)(|vI/«P + |vI/:«P) 

+ ^Ad*?!^ + i^-i^n^ + A'l^'^n^'i'^p, (A4) 



where A = ^(Ai + A2 + 2A12) and A' = ^(Ai -I- A2 - 2A12). 

These fi'ee energies thus show that the energetically 
preferred form of the AMSF state is determined by the 
coefficient A' of last term in Eq. ()A4|) . For A' > 0, that 
is, Al -I- A2 > 2Ai2, the single Q FF-like state is selected. 
On the other hand, for A' < 0, that is, Ai -I- A2 < 2Ai2, it 
is the LO-like state that has the lowest energy. 

Combining the above requirement on A' for the stabil- 
ity of the LO-like state with the condition for two-species 
miscibility, A1A2 > X12, we find an inequality, 

< A12 < v/A^ (A5) 

which for positive couplings Ai can be shown to have a 
zero range of stability. Thus, as advertised, within mean- 
field approximation it is the single Q FF-like AMSF state 
that is always energetically selected. Perhaps the LO-like 
AMSF form can be realized for a metastable atomic gas 
with Ai < 0, as, for example, realized by a ^^Rb-*^Rb 
mixture. 



^ M. H. Anderson, J. R. Ensher, M. R. Matthews, C. E. 

Wieman, and E. A. Cornell, Science 269, 198 (1995). 
^ K. B. Davis, M.-O. Mewes, M. R. Andrews, N. J. van 

Druten, D. S. Durfee, D. M. Kurn, and W. Ketterle, Phys. 

Rev. Lett. 75, 3969 (1995). 
^ C. A. Regal, M. Greiner, and D. S. Jin, Phys. Rev. Lett. 

92, 040403 (2004). 

* M. W. Zwierlein, C. A. Stan, C. H. Schunck, S. M. F. 
Raupach, A. J. Kerman, and W. Ketterle, Phys. Rev. 
Lett. 92, 120403 (2004). 

' J. Kinast, S. L. Hemmer, M. E. Gehm, A. Turlapov, and 
J. E. Thomas, Phys. Rev. Lett. 92, 150402 (2004). 
C. Chin, M. Bartenstein, A. Altmeyer, S. Riedl, S. 
Jochim, J. H. Denschlag, and R. Grimm, Science 305, 
1128 (2004). 

G. B. Partridge, K. E. Strecker, R. L Kamar, M. W. Jack, 
and R. G. Hulet, Phys. Rev. Lett. 95, 020404 (2005). 

* M. W. Zwierlein, J. R. Abo-Shaeer, A. Schirotzek, C. H. 
Schunck, and W. Ketterle, Nature 435, 1047 (2005). 

^ E. Timmermans, P. Tommasini, M. Hussein, and A. Ker- 
man, Phys. Rep. 315, 199 (1999). 

K. Dieckmann, C. A. Stan, S. Gupta, Z. Hadzibabic, C. 

H. Schunck, and W. Ketterle, Phys. Rev. Lett. 89, 203201 
(2002). 

" K. M. O'Hara, S. L. Hemmer, M. E. Gehm, S. R. Granade, 
and J. E. Thomas, Science 298, 2179 (2002). 
T. Bourdel, J. CubizoUes, L. Khaykovich, K. M. F. Ma- 
galhaes, S. J. J. M. F. Kokkelmans, G. V. Shlyapnikov, 
and C. Salomon, Phys. Rev. Lett. 91, 020402 (2003). 

" K. E. Strecker, G. B. Partridge, and R. G. Hulet, Phys. 
Rev. Lett. 91, 080406 (2003). 

R. A. Duine and H. T. C. Stoof, Physics Reports 396, 115 
(2004). 

P. Nozieres and S. Schmitt-Rink, J. Low Temp. Phys. 59, 



195 (1985). 

C. A. R. Sa de Melo, M. Randeria, and J. R. Engelbrecht, 
Phys. Rev. Lett. 71, 3202 (1993). 

M. Holland, S. J. J. M. F. Kokkelmans, M. L. Chiofalo, 
and R. Walser, Phys. Rev. Lett. 87, 120406 (2001). 
1* Y. Ohashi and A. Griffin, Phys. Rev. Lett. 89, 130402 
(2002). 

^® A. V. Andreev, V. Gurarie, and L. Radzihovsky, Phys. 

Rev. Lett. 93, 130402 (2004). 
^° J. Levinsen and V. Gurarie, Phys. Rev. A 73, 053607 

(2006) . 

V. Gurarie and L. Radzihovsky, Ann. Phys. (NY) 322, 2 

(2007) . 

2^ T.-L. Ho and R. B. Diener, Phys. Rev. Lett. 94, 090402 
(2005). 

V. Gurarie, L. Radzihovsky, and A. V. Andreev, Phys. 
Rev. Lett. 94, 230403 (2005). 

C.-H. Cheng and S.-K. Yip, Phys. Rev. Lett. 95, 070404 
(2005). 

J. P. Gaebler, J. T. Stewart, J. L. Bohn, and D. S. Jin, 
Phys. Rev. Lett. 98, 200403 (2007). 

J. Zhang, E. G. M. van Kempen, T. Bourdel, L. 
Khaykovich, J. CubizoUes, F. Chevy, M. Teichmann, L. 
Tarruell, S. J. J. M. F. Kokkelmans, and C. Salomon, 
Phys. Rev. A 70, 030702 (2004). 

J. Levinsen, N. R. Cooper, and V. Gurarie, Phys. Rev. A 
78, 063616 (2008). 

S. L. Cornish, N. R. Claussen, J. L. Roberts, E. A. Cornell, 
and C. E. Wieman, Phys. Rev. Lett. 85, 1795 (2000). 
L. Radzihovsky, J. Park, and P. B. Weichman, Phys. Rev. 
Lett. 92, 160402 (2004). 

L. Radzihovsky, P. B. Weichman, and J. I. Park, Annals 
of Physics 323, 2376 (2008). 

M. W. J. Romans, R. A. Duine, S. Sachdev, and H. T. C. 



31 



32 



Stoof, Phys. Rev. Lett. 93, 020405 (2004). 
V. N. Efimov, Sov. J. Nucl. Phys. 12, 589 (1971). 



3^ S. Basu and E. J. Mueller, Phys. Rev. A 78, 053603 

(2008) . 

J. L. Roberts, N. R. Claussen, S. L. Cornish, E. A. Donley, 
E. A. Cornell, and C. E. Wieman, Phys. Rev. Lett. 86, 
4211 (2001). 

^'^ S. Diehl, M. Baranov, A. J. Daley, and P. ZoUer, Phys. 
Rev. B 82, 064509 (2010). 

S. Diehl, M. Baranov, A. J. Daley, and P. ZoUer, Phys. 
Rev. B 82, 064510 (2010). 

N. Syassen, D. M. Bauer, M. Lettner, T. Volz, D. Dietze, 
J. J. Garcia-RipoU, J. L Cirac, G. Rempe, and S. Diirr, 
Science 320, 1329 (2008). 
^® H. A. Bethe Handhuch der Physik, second edition. Vol. 
24-1, Berlin: Springer (1933), pp. 452-462. 
S. Ejima, M. J. Bhaseen, M. Hohenadler, F. H. L. Essler, 
H. Fehske, and B. D. Simons, Phys. Rev. Lett. 106, 
015303 (2011). 

*° L. Bonnes and S. Wessel, Phys. Rev. Lett. 106, 185302 
(2011). 

M. J. Bhaseen, A. O. Silver, M. Hohenadler, and B. D. 
Simons, Phys. Rev. Lett. 103, 265302 (2009). 
S. B. Papp, Ph.D. thesis. University of Colorado (2007). 
S. B. Papp, J. M. Pino, and C. E. Wieman, Phys. Rev. 
Lett. 101, 040402 (2008). 

A. B. Kuklov, Phys. Rev. Lett. 97, 110405 (2006). 
'^^ W. V. Liu and C. Wu, Phys. Rev. A 74, 013607 (2006). 
*® L. Radzihovsky and S. Choi, Phys. Rev. Lett. 103, 095302 

(2009) . 

J. Stenger, S. Inouye, D. M. Stamper-Kurn, H.-J. Mies- 
ner, A. P. Chikkatur, and W. Ketterle, Nature 396, 345 
(1998). 

A. Gorlitz, T. L. Gustavson, A. E. Leanhardt, R. Low, A. 
P. Chikkatur, S. Gupta, S. Inouye, D. E. Pritchard, and 
W. Ketterle, Phys. Rev. Lett. 90, 090401 (2003). 
M.-S. Chang, C. D. Hamley, M. D. Barrett, J. A. Sauer, 
K. M. Fortier, W. Zhang, L. You, and M. S. Chapman, 
Phys. Rev. Lett. 92, 140403 (2004). 

M.-S. Chang, Q. Qin, W. Zhang, L. You, and M. S. Chap- 
man, Nature Physics 1, 111 (2005). 

M. Vengalattore, S. R. Leslie, J. Guzman, and D. M. 

Stamper-Kurn, Phys. Rev. Lett. 100, 170403 (2008). 
52 T.-L. Ho, Phys. Rev. Lett. 81, 742 (1998). 
5^ T. Ohmi and K. Machida, J. Phys. Soc. Japan 67, 1822 

(1998). 

^'^ D. M. Stamper-Kurn, M. R. Andrews, A. P. Chikkatur, S. 
Inouye, H.-J Miesner, J. Stenger, and W. Ketterle, Phys. 
Rev. Lett. 80, 2027 (1998). 

M. R. Matthews, D. S. HaU, D. S. Jin, J. R. Ensher, C. 

E. Wieman, E. A. Cornell, F. Dalfovo, C. Minniti, and S. 

Stringari, Phys. Rev. Lett. 81, 243 (1998). 

C. K. Law, H. Pu, and N. P. Bigelow, Phys. Rev. Lett. 

81, 5257 (1998). 
" T.-L. Ho and S. K. Yip, Phys. Rev. Lett. 84, 4031 (2000). 
5* F. Zhou, Phys. Rev. Lett. 87, 080401 (2001). 
5^ E. Demler and F. Zhou, Phys. Rev. Lett. 88, 163001 

(2002). 

^° R. Barnett, A. Turner, and E. Demler, Phys. Rev. Lett. 
97, 180412 (2006). 

S. Mukerjee, C. Xu, and J. E. Moore, Phys. Rev. Lett. 
97, 120406 (2006). 
"2 J. Mur-Petit, M. Guilleumas, A. Polls, A. Sanpera, M. 
Lewenstein, K. Bongs, and K. Sengstock, Phys. Rev. A 



73, 013629 (2006). 

A. Lamacraft, Phys. Rev. Lett. 98, 160404 (2007). 
^'^ S. Mukerjee, C. Xu, and J. E. Moore, Phys. Rev. B. 76, 
104519 (2007). 

S. Foiling, A. Widera, T. MuUer, F. Gerbier, and I. Bloch, 
Phys. Rev. Lett. 97, 060403 (2006). 

G. K. Campbell, J. Mun, M. Boyd, P. Medley, A. E. Lean- 
hardt, L. G. Marcassa, D. E. Pritchard, and W. Ketterle, 
Science 313, 649 (2006). 

R. A. Barankov, C. Lannert, and S. Vishveshwara, Phys. 
Rev. A 75, 063622 (2007), and references therein. 
A. F. Andreev and I. M. Lifshitz, Zh. Eksp. Teor. Fiz. 56, 
2057 (1969) [Sov. Phys. JETP 29, 1107 (1969)]. 

G. V. Chester, Phys. Rev. A 2, 256 (1970). 

™ A. J. Leggett, Phys. Rev. Lett. 25, 1543 (1970). 

E. Kim and M. H. W. Chan, Nature 427, 225 (2004); 
Science 305, 1941 (2004). 

^2 P. G. de Gennes and J. Prost, The Physics of Liquid Crys- 
tals (Oxford University Press, 1995), 2nd ed. 
P. Fulde and R. A. FerreU, Phys. Rev. 135, A550 (1964). 
A. I. Larkin and Y. N. Ovchinnikov, Sov. Phys. JETP 20, 
762 (1965). 

Y. Liao, A. S. C. Rittner, T. Paprotta, W. Li, G. B. Par- 
tridge, R. G. Hulet, S. K. Baur, and E. J. Mueller, Nature 
467, 567 (2010). 

A. Bianchi, R. Movshovich, C. Capan, P. G. Pagliuso, and 
J. L. Sarrao, Phys. Rev. Lett. 91, 187004 (2003). 

"^"^ T. Mizushima, K. Machida, and M. Ichioka, Phys. Rev. 
Lett. 94, 060404 (2005). 

D. E. Sheehy and L. Radzihovsky, Phys. Rev. Lett. 96, 
060401 (2006). 

D. E. Sheehy and L. Radzihovsky, Ann. of Phys. 322, 2 
(2007). 

*° L. Radzihovsky, D. E. Sheehy, Rep. Prog. Phys. 73, 
076501 (2010). 

H. J. Kimble, Quantum fluctuations in quantum optics 
- squeezing and related phenomena, les Houches, Session 
LIII, 1990 (Elsvier, 1992) and references therein. 

*2 C. Orzel, A. K. Tuchman, M. L. Fenselau, M. Yasuda, 
and M. A. Kasevich, Science 291, 2386 (2001). 
L. Radzihovsky and A. Vishwanath, Phys, Rev. Lett. 103, 
010404 (2009). 

L. Radzihovsky, Phys. Rev. A 84, 023611 (2011). 

For unequal masses FR interaction is given by Hfr = 

a<^t . \ mi ^ E12_^ (_jv)^il + h.c. as 

required to preserve Galilean invariance. 
C. A. Regal, C. Ticknor, J. L. Bohn, and D. S. Jin, Phys. 
Rev. Lett. 90, 053201 (2003). 
^"^ L. D. Landau and E. M. Lifshitz, Quantum Physics 
(Butterworth-Heinemann, Oxford, 1980). 

B. D. Esry, C. H. Greene, J. P. Burke, Jr., and J. L. Bohn, 
Phys. Rev. Lett. 78, 3594 (1997). 

Y. W. Lee and Y. L. Lee, Phys. Rev. B. 70, 224506 (2004). 
M. P. A. Fisher, P. B. Weichman, G. Grinstein, and D. S. 
Fisher, Phys. Rev. B. 40, 546 (1989). 
J. W. Negele and H. Orland, Quantum Many-Particle Sys- 
tems (Addison- Wesley, 1988). 
^2 S. Sachdev and N. Read, Int. J. Mod. Phys. B 5, 219 
(1991); S. Sachdev, Lecture Notes at the Les Houches 
Summer School on "Modern theories of correlated elec- 
tron systems" (2009), Lecture Notes at the Mahabalesh- 
war Condensed Matter School, International Center for 
Theoretical Sciences, India (2009), arXiv: 1002. 3823 



32 



Chao-Ming Jian and Hui Zhai, Phys. Rev. B 84, 060508 

(2011). 

®^ For simplicity it is convenient to first consider a homoge- 
neous bulk system or a "box" trap, and take into account 
the trap-induced local inhomogeneity using LDA, valid for 
large condensates and smooth trapping potential. 

®® In the absence of an optical potential or quenched disor- 
der that can "locaUze" atom number and induce Mott- or 
Bose-glass insulating phases, only Bose-condensed, super- 
fluid ground states are expected at zero temperature. 

^® L''Ar(l) and ?7AAr(l) symmetries are associated with the 
total atom number N = Ni + N2 + 2Nm and difference 
AN — Ni — N2 conservations, respectively. 

^"^ More accurately, it is not g2 but T-matrix, i.e., scattering 
length a's that matter. 

^® In the context of conventional spinor condensate, the po- 
lar state is also termed "nematic" or "antiferromagnetic" 
superfluid. 

®® P. M. Chaikin and T. C. Lubensky, Principles of Con- 
densed Matter Physics (Cambridge University Press, 
1995). 

To keep the form of the Landau free energy simple, one 
needs to be sufficiently close to the transition boundaries 
(but staying outside the critical region, where local Lan- 
dau mean-field expansion fails) to allow a local expansion 
in the order parameters. 

K.-S. Liu and M. E. Fisher, J. Low Temp. Phys. 10, 655 
(1973). 



J.M. Kosterlitz, D.R. Nelson, and M. E. Fisher, Phys. 
Rev. B 13, 412 (1976). 

P. Calabrese, A. Pelissetto, and E. Vicari, Phys. Rev. B 
67, 054505 (2003). 

I. Bloch, J. DaUbard, W. Zwerger, Rev. Mod. Phys. 80, 
885 (2008). 

B. I. Halperin, T. C. Lubensky, and S. K. Ma, Phys. Rev. 
Lett. 32, 292 (1974). 

S. Coleman and E. Weinberg, Phys. Rev. D 7, 1888 
(1973). 

^"^ D. Podolsky, S. Chandrasekharan, and A. Vishwanath, 
Phys. Rev. B 80, 214513 (2009). 
E. Babaev, Phys. Rev. Lett. 89, 067001 (2002). 
N. D. Mermin, Rev. Mod. Phys. 51, 591 (1979). 
B. DeMarco, C. Larmert, S. Vishveshwara, and T.-C. Wei, 
Phys. Rev. A 71, 063601 (2005). 

M. Greiner, O. Mandel, T. Esslinger, T. W. Hansch, and 
I. Bloch, Nature 415, 39 (2002). 
"2 M. W. Zwierlein, A. Schirotzek, C. H. Schunck, and W. 

Ketterle, Science 311, 492 (2006). 

G. B. Partridge, W. Li, R. 1. Kamar, Y. Liao, and R. C. 
Hulet, Science 311, 503 (2006). 

Y. Shin, M. W. Zwierlein, C. H. Schunck, A. Schirotzek, 
and W. Ketterle, Phys. Rev. Lett. 97, 030401 (2006). 
"® S. Nascimbene, N. Navon, K. J. Jiang, L. Tarruell, M. Te- 
ichmann, J. McKeever, F. Chevy, and C. Salomom, Phys. 
Rev. Lett. 103, 170402 (2009). 



